A Well Rounded Engineer

It is that time of year once again where everyone's favorite question - "Do you have any New Years Resolutions?" - is being asked around the dinner table, at social gatherings, and even at the workplace. Whether you are someone who spends time developing a thorough list of goals and strategies, or have given up on the idea of resolutions altogether, I think the underlying question is probably still at the forefront of our minds - "How can I become more like the person I want to be?"

For many engineers, myself included, the answer to this question usually includes goals around improving our technical skills.  Usually this is because we place value in being technically competent and, maybe more importantly, we just enjoy doing technical things. But if your ultimate goal is to become a better engineer, to solve bigger problems, have a larger impact, ship things that matter, and grow in your career, then I would propose that it may not be your technical skills that are holding you back. Strong technical skills are the table stakes for being an engineer, but the truth is that the things we build don't exist in a vacuum. Most engineers plateau not because they hit their technical ceiling, but because the truly impactful problems that need solving are not purely technical. A truly effective engineer is one who is well rounded enough to be able to see, understand, and solve problems at the intersection of technical, social, and organizational systems.

 The path to becoming a more well rounded engineer will ultimately be unique to each individual, and not one that I would want to try and fit into a single blog post. I will, however, offer a framework that has worked well for me to help shape and prioritize how I approach this goal. The core belief that underpins this approach is that one must strive for a healthy mind, body, and soul (or psyche) and that a deficiency in any one impacts all the others. Applying this to our goal of being a more well rounded engineer, I try to focus on spending time doing things that present me with different viewpoints (mind),  challenge my physical limits (body), and push me out of my comfort zone by creating things outside of my domain (soul).

If this is resonating with you so far, then I would suggest starting by setting a reasonably short time frame, 2 to 4 weeks, and carving out some intentional time for the following:

 

Do something that makes you think

Reading, listening to music, watching movies; these are all things that don't have to be just mindless consumption of media. We have unprecedented access to so much good content that can present us with new ideas, challenge our existing assumptions, and force us to consider new or different viewpoints. 

Engineers often spend much of their day thinking, so it can be easy to default to thinking about technical topics, but I highly recommend picking something well outside of the technical domain. Business books can be a good starting point, but some of my most thought provoking and valuable insights have come from philosophy, politics, memoirs and other sources well outside the engineering domain.

Taking the time to really understand viewpoints well outside our own experience strengthens our frameworks for how we approach and interact with our social and political (i.e. human) systems and allows us to more effectively communicate and align interests across the organization.

 

 Do something that makes you move

There are so many physical and psychological benefits to maintaining even a moderate level of physical activity. This is one that many engineers will acknowledge theoretically, but either don't prioritize it, or have actively bought into the lie that real engineers aren't athletic. 

For the longest time I found myself in this camp, and I thought that because I wasn't good at sports and didn't have the motivation to lift weights or go running, that I just wasn't an active person. The key insight for me came when I realized that 1) physical activity is more about discipline (getting out even when you don't feel like it) than it is motivation, and 2) there are so many other kinds of activities outside of the traditional 'working out' and not all of them have to be competitive. Once I found a set of activities that I truly enjoyed (hiking, mountain biking, skiing) I all of the sudden found I was making more and more time to go do those things. If all this still sounds like a far reach, don't be afraid to start small by just taking a walk on your lunch break.

The more time I spend being active, the more I realize it is just as much a mental game as it is physical. The ability to push ourselves, find our limits, and prove to ourselves that we can do hard things is something that will carry over to your professional life. Those impactful and big engineering challenges will rarely be ones that can be solved quickly and easily and staying active gives us the discipline, energy, stamina, mental clarity, and resilience to tackle those problems with confidence.

 

Do something that makes you create 

 I've met many engineers who are wonderful artists and musicians, so for them this advice probably comes very naturally. But if you are one who has so far focused mostly on technical products,  I can't recommend highly enough the benefits of creating something outside of the technical domain. 

The act of creating is woven deep into our being, but in our highly connected age it can be difficult to just enjoy the act of creation without worrying about how it compares to everyone else, or how it could be marketable as a side hustle. It can be hard to put yourself out there and expose yourself to criticism, especially when it involves doing something new. The thing is, so often I've found that the most criticism comes from myself. By forcing myself to quiet that inner critic and get past my fear of failure, I find that I end up exercising a creative muscle that directly benefits my general problem solving skills.

 

Rinse and Repeat

I can't promise that if you do these things you will all of the sudden find yourself getting that next job or promotion. But if over the next couple of weeks you do it, and then do it again, and again, eventually you will find some probably quite unexpected things that you truly enjoy doing. Having invested in broadening your horizons and skills you will be better equipped to navigate both the technical and the non technical systems that comprise so many of the more interesting problem sets. And maybe some day down the road you'll be presented with one of those problems (opportunities) that connects a diverse set of domains that you are uniquely equipped to provide insight into and solve. 

That is what I think makes for a great engineer.

Developing Successful Demos

Over the course of my career I've had the opportunity to lead or be in the audience for dozens, if not hundreds, of technical demonstrations. I would assume that this is actually fairly typical for many engineers in the tech industry, and yet despite this, so many in our field find the act of leading a demo to be difficult. Even if you haven't had to lead a demo, it is quite likely that you have at least sat through a bad demo. Maybe the speaker was not engaging, or maybe it was uninformative, or most likely the product just didn't work. 

So why are technical demonstrations, something that is so common in our industry, so hard?

I don't think the answer is all that surprising: Most engineers like building things and aren't necessarily skilled or trained in public speaking. Most demos are given in non-ideal environments. Most products are still under active development which can sometimes lead to breaking changes or missing features. etc., etc., etc.

Rather than try and solve all of these issues, because I don't think you truly can get rid of 100% of them, the advice I will offer instead is that we re-frame how we view technical demonstrations. A demo is, at its core, a story about your product. Most people focus solely on having a good product (and for good reason!), but a good story forms the foundation for a good demo, and without it no matter how good the actual product is, the demo will not be memorable or effective.

Give enough demos, and eventually you will forget a line, or a critical feature won't be ready on time, or something will break. There's no amount of planning and scripting that will prepare you for every eventuality. But if you develop a compelling story, and keep laser focused on telling that story, then you can more easily adapt and improvise to still give a successful demo in spite of whatever issue you face.

How exactly to develop a compelling story is something people much better than myself spend their entire lives trying to learn, so while I'll leave most of that to the experts, I will recommend starting by answering some framing questions:

• What are you selling? It seems a bit obvious, but really try to take the time to understand the product itself, what differentiates it from the competition, and what it is offering the buyer. Keep in mind that sometimes the product you want to sell is not actually the tangible thing you are showing, but rather the person or team or organization behind that thing.
• Who are you selling it to? Are they prospective buyers, existing customers, your own boss or company leadership? Are they primarily technical, or more executive? Are they the ones who will actually be using the product? Ultimately you are asking what do they care about and what will resonate with them?

Once you have established the 'who' and the 'what' you can then approach your demo by thinking about some of the general principles that make for a good story. 

• A story is not just a list of facts. Of course facts are a good thing, and lend credence to your product, but they are meant to support the story, not the other way around. This is NOT saying you should lie, quite the opposite, but if you find yourself listing things like supported protocols then ask yourself if this is adding to the story, or if it can be skipped and instead refer someone to the documentation.
• A story is not a 'how-to' guide. There may be elements of how easy it is to accomplish something in your demo, but if someone needs a detailed step by step process guide then they can refer to the documentation.
• A good story is relatable. Even if what you are showing is a critical differentiator and nothing goes wrong, if your demo is not relatable to the audience then they will likely walk away with a 'so what?'. I recommend developing a comprehensive story that hits on all major points or features and then tuning each demo with only a subset of those 'chapters' based on each individual audience and what they care about.
• A good story is singular and self contained. It can be tempting to try and pack in several different threads in an attempt to throw spaghetti at the wall and see what sticks. The problem is that doing this can lead to overly long and complex demos that leaves the audience remembering a bunch of moments, but not the main point of your story because it got lost in the noise.

If you've established a clear plot thread in the audience's mind, then it doesn't matter quite as much if the button you click actually does the thing it is supposed to do, or if you just tell them what it would do if it were implemented. You can more easily adapt and improvise because you know the framework of the story. In fact, I would argue that with a compelling story to your demo, you can be 100% up front and truthful about what works and what doesn't, and it won't matter.

 

We humans are wired for telling and listening to stories. Our brains will more easily remember stories, and can automatically fill in gaps if needed. The iconic Star Wars bounty hunter, Boba Fett, has approximately 6 minutes of total screen time and only 4 lines in the entire original trilogy. Despite this, he became a massive fan icon generating a significant lore and still remembered today. A good story will draw people in, be memorable, and leave them wanting more. Whether it is a demo, a presentation, a report, or anything else in your job requiring communication, stop and ask yourself what is the story I am trying to tell?

 

CandyOps: Over-Engineering Halloween with AI and Robots

A few weeks ago my wife sent me a link to this candy throwing robot and said it looked like a fun project for our kids' upcoming Trunk or Treat. I of course immediately took that as permission to buy a new Raspberry Pi5 and all the necessary parts (including a few unnecessary ones) for the SO101 Robot Arm under the thinly veiled excuse of a great father/daughter project (and totally not for my own amusement).

Project Goals

"We do these things not because they are easy, but because we thought they would be easy." 

As with most personal projects, the end state and project goals have evolved over time. I originally set out to just replicate the linked candy throwing robot project, but we quickly decided we would get more out of it if we tried to build our own thing as opposed to copy/pasting someone else's project. 

At a high level, my goals for this project were:

• Share the learning and discovery process of building something with my kids.
• Automate the act of picking up a piece of candy (ideally out of a bowl) and hand it to a trick or treater. 
• Automate the triggering of the hand-out process through some visual or audio input.
• Build a portable assembly that could be moved easily and did not require internet access.

Assembly

I opted to buy the motor kit rather than individually sourcing each part and we 3D printed the parts using this model that is optimized for our printer. I also picked up a Raspberry Pi 5 to control everything, although given the amount of AI models and training we tested for this project I do wonder if the Jetson Orin Nano would have been a better choice.

Printing the robot linkages

 Other than a few self-inflicted issues, the printing and assembly were pretty straightforward. I also added the overhead and wrist mounted cameras to better support the data collection for imitation learning.

Assembling the robot

Once assembled, we followed the instructions for motor configuration and calibration. I did manage to burn out one of the motor boards during calibration when a cable came loose and shorted the circuit, so I highly recommend double checking the motor cables are fully seated and run it through the full rotation BEFORE applying power.

After a quick test to make sure everything worked, we mounted the RPI, power strip, and both of the arms to a scrap piece of plywood for stability and to make it easy to move around. This has the added benefit of giving a very consistent environment when recording and training the models.

Fully assembled robot and work area

Training

 I knew from reading the LeRobot Documentation, and my own personal experience with projects like these, that going straight to the complex imitation learning through neural networks was going to cause a lot of headaches. Luckily, they provide a nice guide that sets out a series of incremental steps that would help us get more familiar with operating the robot, and at the same time provide nice gates that, if nothing else worked, would still give a fun experience for the Trunk or Treat. 

Manual Operation

The first step was just to get the robot working with manual teleoperation. In this scenario the user controls the Leader arm (the one on the right with the handle) and the Follower arm imitates the movements. LeRobot provides some handy scripts to get this all working and in no time we were able to practice moving the robot around. Honestly, we could have stopped at this point and the kids would have been super happy just trying to guide the robot to pick up candy out of a bowl.

 

Manual Tele-operation of the robot

Record/Replay

The next step was to practice making a recording and then replaying that recording. In this scenario you operate the robot much like you do with manual teleoperation, except this time the software is recording all of the servo positions over time into a dataset that can be played back later in the future with no manual intervention. We practiced recording the arm reaching into the bucket and handing someone some candy and then replaying that recording. This mode of operation proved to be the most reliable and was easy to string together with some basic control logic to replay specific episodes based on desired conditions.

Episode Replay with Motor Positioning Data

 

Traditional Kinematics

The traditional way of controlling robots would be to model the physical robot and then develop some forward and inverse kinematic solvers. This would allow you to provide it a set of normalized coordinates and generate the necessary motor positions, or provide it a set of motor positions and know where it is in coordinate space. The LeRobot  and SO101 library come with some helpful tools for this (URDF models, Mujoco for simulation, etc), but we skipped this mode entirely as it felt a bit too complex for a first robotics project. I do plan to revisit this for future projects.

Neural Network 

Because LeRobot is a Hugging Face project, it has a lot of scripts and utilities for generating datasets and training those datasets to build a neural network. The neural net can then be fed into the Robot Processor Pipeline to take observations (motor positions, camera feeds) and infer future state. This is definitely the most interesting mode as it has the potential for a lot of flexibility for operations like Pick and Place of an object. 

Wandb output of our first training run

We made two attempts at this process - the first one was with the full candy bowl. This turned out to have way too many variables and the resulting model did not work at all. The second attempt was a much more pared back pick and place with a single candy. This one got closer, but the movement was still very jittery, and it often missed the candy all together. I will say, it was very cool to see it miss the candy, start moving away, realize it wasn't holding the candy, and then go back to try again.  

One of the rare successful attempts using our neural net model

We had a lot of fun making the datasets, and brainstorming why it was exhibiting certain behaviors after training, but because of the slow process for creating datasets, training (took about an hour on an A100 I rented), and actual pickup time we ultimately decided not to use this mode for our Trunk or Treat. 

My initial sense is that just like traditional LLMs, the premise here is very exciting for the field of robotics, but the scale required for datasets and training is probably beyond the reach for all but the largest of organizations.

Putting it all together

To this point we had been mostly controlling the robot directly from a laptop by running scripts manually. We knew this wouldn't be very useful in a Trunk or Treat setting, so we set out to tie it all together. For this part I wanted my daughter to start dipping her toes into coding, so I set up VSCode with Copilot and walked her through some basics of AI assisted coding. 

Our first thought was to have the robot detect a person through the camera, and then run the pass out candy replay. We tested out a small YOLO model which was surprisingly easy to get working in python, and ran really well on the RPI5 (about 1 - 2 Hz). We quickly decided this would be difficult in a real life setting as there are often dozens of people walking around during a typical Trunk or Treat and so it would likely ALWAYS be detecting a person.

Our second attempt was to use a voice command. Again, I was surprised at both how well copilot took some very basic requests and turned it into working code, as well as how well the Vosk voice models ran on the RPI. This mode seemed to work better as we could give it specific commands to wait for, as well as piece together different replays based on different commands.

Finally, we worked with copilot to build a web frontend so we can control the robot from a cell phone. We ended up with a simple Flask based python web frontend, and even got it themed for Halloween.

Command and Control interface

 At this point, I took over to clean up the AI generated code and make it a bit more robust:

• The first iteration was simply calling lerobot scripts for replaying a dataset episode. This worked fine, but we found there was a few second delay as the script set everything up and loaded the dataset. I updated the code to do all of these loading and setup steps during the initialization function so that each replay call was much more responsive.
• The voice model worked OK, but had a lot of false positives which would only get worse in a crowded noisy environment. I found that the KaldiRecognizer class could be given a list of strings and only register those in the model. This greatly reduced the false positives and seemed much easier than finetuning the model.
• Not so much a code change, but I found during the Trunk or Treat that it was pretty easy for me to naturally work in the registered key words ('Trick or Treat', 'please' etc) when talking to kids if it didn't register right away. This made it feel natural even when the voice activation didn't work as expected. 

Conclusion

Just like any project, whether DIY or professional, there comes a point where there are still 100 things you would like to do and about 10 times as many ways it could all go sideways, but time marches on and deadlines come quicker than you want. Fortunately, we had tested and planned enough that our candy passing robot was a huge success at the Trunk or Treat event.

Trunk or Treat!

Whether it is your kid, your mentor, or the new junior on the team, I believe there is value in not just teaching the next generation what you already know, but also finding opportunities to experience the learning process together. We both had a ton of fun on this project, and my hope is that I've been able to move the needle from "my dad knows how to build robots" toward "my dad and I figured out how to build a robot and I'll bet if I had to, I could do it again".