This is a rush transcript. The first 40 minutes of the 2 hour 40 minute episode have been edited for clarity. The remainder is an automated translation. A complete transcript will be available no later than one week after release at symbolcrash.com.
Ben: Hello, and welcome to the next edition of the Hack the Planet podcast, the only podcast in the comedy category that averages less than one joke per episode. Well, there wasn't a bitter sarcasm category. So I, I got a little confused by the ontology. We are joined today by legendary hardware security trainer, Joe Fitzpatrick.
Although the Patrick is usually silent, say hi, say hello, Joe.
Joe Fitz: Hey,
Ben: uh, we're also joined by, uh, our usual panel. We have Mitchell, Max , have Vyrus and we have Justin.
Justin: Hey guys. It's good to see you.
Ben: Hey, so, uh, Joe, uh, how have you been?
Joe Fitz: Pretty good. Uh, surviving, enjoying the fact that I haven't traveled anywhere in nearly a year and a half or over a year and a half at this point. So kind of a different pace from what I'm used to.
Ben: Yeah, so normally, uh, you're, you're going around, uh, doing hardware security trainings all over the place. Like what's your normal travel schedule.
Joe Fitz: Um, I try to keep it to one, uh, trip a month. Um, I used to in the normal days and, uh, I'd never really did a good job of that. So it usually ended up being. One and a half to two. And, uh, so sometimes when one of those trips would be two weeks of training back to back, it, it got pretty hectic. Yeah. I've actually on my, my second time that my office had been clean ever both times in the past year and a half, because I don't, I don't get home, uh, drop the bag and start packing the new bag and then come home and drop the bag and start packing a new bag.
And then, you know, finally get home and have a month where I didn't book anything. And I have like eight bags that I have to go through to unpack.
Ben: So you weren't, I, you, weren't always this busy, like a, uh, you know, a few years ago, how long have you been doing the, the security training thing?
Joe Fitz: So I, uh, this is nine years now that I have been independent running my business, doing this. Um, and it's pretty interesting cause you know, it, it started off like any business starts off, relatively slow and has built up over the years. Um, but 2019 was like, yeah, 2018 or 2019 where we're at at capacity.
And I kept wondering like, oh, is this what I'm supposed to back off? This is where I'm supposed to back off, but it's still going good. Um, and luckily the world decided it was time to back off in 2020 and I was ready for it. So complaints there.
Ben: So how, uh, how did you, how did you get into hardware security training in the first place? Like, it's a, it's a, it's a relatively small niche. It's like you and,
Joe Fitz: Yeah.
Ben: Joe Grand and
Joe Fitz: Colin O'Flynn, few others. yeah.
Um, so I was working at Intel. Um, I started out in the product, uh, uh, debug, sorry, the silicon debug group and moved into the product security group. And, um, don't know that I was actually very good at what I was doing at that point in time. But that's, that's an aside.
We started needing to train a whole bunch of people. So everybody who was a functional validator for silicon needed to understand what a security bug was, because if they didn't, they would just file their bugs and ignore them. Um, and there was a small group of us, the product security team, and we were supposed to go and look through and find all the, like, look through all the bugs that are filed and figure out which ones were security relevant.
So what we started doing is training them, training the entire like validation group to identify what a security bug looked like. Um, that way they could click a click a button in, in their bug bug report filing that said this might be security impact. And then instead of looking through thousands of bugs, we looked through a few hundred bugs that have that flag checked.
Um, so I Was uh, I didn't make that training material. One of my colleagues did, but I really enjoy teaching it Um, and when you actually have a course, that's a required course required corporate training and to get people to have. It's the end of the class and be like, wow, that was a good use of my time.
Um, that I think is an achievement. Right? And I, I was able to do that and what I wanted to do that I want it to be doing hands on training. Cause this is a purely lecture based class and there's so much to hardware that you can be doing. Hands-on um, and you know, I had been at Intel for long enough that it was time for me to, to, to look at different directions.
And I decided I, uh, I was on parenting leave. Um, I went to my first Def con and I went into the hardware hacking village and saw the state of hardware hacking, which you know, was great. You know, there's a whole bunch of people who are doing hardware hacking, but the, it was limited. It was, it was soldering, it was flashy lights.
It was some basic stuff. And I'm like, there's so much more we can do here. Um, and that's when, kind of the whole idea congealed, um, and you know, within, within two months of my first Def con I guess I had, uh, uh, left my job and started a business. Um, and I was getting a workshop in a hardware hacking village the next year.
And then the following year I was getting classic black hats. So like one thing led to another pretty quickly.
Vyrus: Was it really different? Like teaching. I mean, I imagine you get a fair amount of software, people who are like, in some of these hardware classes now. And I mean, like you mentioned earlier, like being at Intel where like you have these people who just do validation, like that's a normal thing. And like, there's a little bit of that in software, but like not like there isn't hardware, right?
Like, no, you validate stuff. Right. And so you're like more teaching, like QA people, how to do security as opposed to teaching security people, how to do security different. Like, does that, like this that's gotta be weird because like software does not get tested like hardware.
Joe Fitz: And it's funny because I, you know, even the classes I teach now, I try and push them to saying that like, yeah, this is, this is for that realm between you have hardware people, and you have software security people, and there's a gap in the middle. And so I want to take all the hardware people and bring them in and give them the security skills.
I want to take the software people and bring them in and give them the hardware skills. And so I can use the same training material for both of those audiences. So as long as I ask him ahead of time, I, you know, I, I include the right anecdotes and tangents and it comes to a nice spot in the middle. Um, and the group I was working with at Intel was a lot of software security people, uh, pulled into the hardware. and I was one of the oddball hardware.
People kind of pulled into the security side of things. Um, so you know, it, it, there's both groups of people and the, the, the, the savvy hardware, hackers and hardware security people come from both sides. Um, so yeah, it's different, but it's, it's not that different.
Vyrus: That's cool. It seems like it'd be harder. Cause like hardware definitely scares me,
Joe Fitz: it's like that Venn diagram of like, you know, what you know, and what people, you know, know, like you're just finding the overlap or the finding the gap and the Venn diagram and filling it in.
So it doesn't make a big difference, which direction from.
Vyrus: I might have to, I'd have to take the plunge and take one of these trainings Sunday. Cause it definitely seems really daunting as like a software person. Cause I think about. How big the Venn diagram of just software is. And it's like on the one hand you have like a waspy stuff. And on the other hand, like the far end of that pool, you get into like loaders and debuggers and craziness.
Right. And it's like, when you're at the debugger part and you're, you know, staring like somebody who's just starting in the, in the face and they're like, I want to do that. It's like, you want to give them all the things, but you also don't want to scare them. Right. Cause like, there's this giant chasm. And like, it seems like, and I don't know this.
Right. Cause I'm not a hardware person, but like, it seems like on hardware that has to be like the same problem. Only like five layers deeper. Right. Cause it's like, you're not just teaching hardware. You're also teaching at some level like physics and like a little bit of chemistry and like, oh, oh goodness.
Ben: It isn't really, it isn't really big space, but there's a lot of, kind of easier to access stuff. Uh, on the surface that gets used a lot more frequently, like, uh, making a serial connection to something, uh, or like using J tag or whatever, versus, um, you know, like, erm, probe side channel analysis or whatever, uh, on the high end or like laser fault injection or whatever the cool kids are doing these days.
Um, I am kind of curious, like how, um, kind of the, the courses you offer the, and the, the sort of the, the content of the courses has evolved over the last nine years.
Joe Fitz: So at the very first workshop I did was, uh, uh, like beginners, you know, side channel attack on a key pad entry system. Um, that's the workshop I did at Def con heat years. ago at this point. Um, and basically you're using a logic analyzer, which is a hardware tool. Um, you know, once you understand how to use the logic analyzer, you realize it's like, it's an observation tool.
It lets you watch things, um, and then capture data and you measure it and you can draw your conclusions. Um, pretty interesting. I'm just still thinking about the whole like software hardware gap, but what actually turned out to be the most, uh, alluring was basically the, the, the doubly hardware hacking 1 0 1, like what can we do to get your hands on hardware, find things and hookup wires to them.
And that's like finding art ports, finding spy, finding dumping, firmware, getting jaytag. And so even though like my prior background was like Silicon debug, you know, like where we had a lab with several fit machines, so we can go and edit, edit ships to go and run on a system while we examined them with, uh, we hit them with a laser to do, you know, manipulations, right.
That's what I've been doing for years. And suddenly I get out and, um, I'm teaching people like you, arts and J tax, which I really enjoy. Um, it's the introductory stuff. And it's also like the gateway. I didn't realize how much a need there was for that gateway. Um, so that's great. Um, it keeps me a hundred percent busy.
I keep trying to make these advanced classes and they're great. And I think people enjoy them, but like, no matter how much effort I put into advanced classes, that one introductory class is 50 to 90% of my work because of the number of people who want and need to go through it. You have someone with a strong software security background and they take that into an introductory class.
It like opens the door for them, and then they just dive in and they can do all sorts of things. They don't need the advanced classes, right. Because they've got the stepping stone, they've got the foundation to work on. Um, it's the people actually more of the hardware side of people who need the more advanced classes because they, we, they need to understand the synchrony applications of the greater, uh, greater issues.
Ben: That's actually, that's a really good point because you were like my, uh, my, my, my role model for, uh, for training in general, but also, um, hardware training specifically. Um, and so, uh, you know, sometime after, uh, so I went through one of your trainings at one point down in Portland years ago, and then sometime later, um, I ran into you at a Def con the first year Hari did the voting machine village.
Uh, and we were hacking, we were hacking one of the voting machines and, uh, it turned out to be an arm five and no one had a dongle that, uh, that worked on anything less than an arm seven. So I was like running around Def con getting blisters, looking for anyone that had an Open OCD, uh, compatible dongle.
And, and I caught you, uh, still wheeling your carts around from your hat training. And you were like, here, have this, uh, uh, FT232 dongle. And then we went and dumped the firmware of the voting machine, which was the first time, uh, someone had dumped that firmware. We found a bunch of bugs in it. It was a very good time, but I went home after that and I started doing this, uh, hardware hacking workshop, which I just put on meetup and I don't charge anybody on it.
I've been doing it once a month ever since, um, and people show up and they're just total beginners and I'm like, Hey, would you like to learn how to hack a voting machine? Like you can totally, like, I can get you, uh, up to getting JTAG and a voting machine inside of the, your first session, you know? And I always thought like, Hey, if, if people keep going back, like we could eventually get up to some more advanced stuff, but it kind of never really, sometimes there's smaller groups that come that are more advanced, they go off and do something else, but it's, it's really like everyone who shows up really just wants that kind of, um, a beginner.
Like how do I get a serial connection? How do you know, how do I get through using Open OCD for the first time, which is traumatic and it's good to have guidance.
Vyrus: Yeah.
Ben: It is, it is a terrible interface. It was written by someone that hates people. But what would you, what you were saying is that like, uh, you know, the, the, if you take a software security person, um, and you show them how to make a UART connection to a board, um, odds are, they're like, oh, Hey, it's a root shell.
Like, I don't need you anymore. Like, I'm going to go. I'm good. I like, I know what to do with the root shell. Like now we're back in, you know, my happy space.
Vyrus: I basically have that exact experience, like, like the very, like the very first hardware thing I ever did was good speed. Teaching me how a good fat worked. And like I used it and I got firmware and then I was like, oh, now I have binary. Oh, I know how this way.
Joe Fitz: I know root shell.
Max: Yeah.
Vyrus: Yeah. a bunch of, yeah, exactly.
Ben: Well, that was actually a direct quote from someone at the last workshop. Right? Like they, they got it, they got a shell finally on a, on a DSL modem and they're like, oh, Hey, a root shell. And they were like, started tuning everything out. And they were like, just typing to now. It's like, I know what to do with this.
Joe Fitz: One fun part you get to though is you get the people who have done a lot of stuff on like full stack Linux operating systems and Windows operating systems or anything else. And you give them a root shell and like, oh, I got a root shell. I can do all this stuff. And they start typing in their stuff.
And like all the, all the flags they use on Netcat are gone,
Ben: Yeah.
Joe Fitz: BusyBox NetCat, then they're like, wait, wait a minute. And then like, oh, let me just enter the file. And like, if there's no text editor and not even VIM works. And they're like, wait, how, uh, you know, I then of course they know how to get around these things.
It's just, you know, they get so used to the set of core tools. They're used to that. Like suddenly you realize, oh, embedded, you don't necessarily have much, you have you have stuff. You have to go and use a shell redirection to rewrite your files because it's just easier.
Vyrus: the joys behind the phrase, what is BusyBox it's like, oh.
Ben: But like, yeah. What is What the fuck is BusyBox? These are all soft links, what the what's going on,
Max: The
one that gets me is just LS never featureful enough on BusyBox. Oh, and PS. Oh man. It only does
Unix or the BSD options.
Vyrus: Or my favorite is, I mean, I was messing with SSH and I don't know what DropBear is, and oh, you're in for some fun.
Ben: But like U-Boot and BusyBox are all like, people are like, oh, it's U-Boot? I know U-Boot. And it's like, yeah, but it's, uh, you know, U-Boot and BusyBox are just giant switch statements that are like configured by a config file that everybody just goes in and like comments in or out, whatever it is they want.
Um, and then they make horrible customizations. What?
Joe Fitz: And customizes and doesn't rerelease
Max: Oh yeah.
Ben: Uh, so it's like, yeah, it's, it's pretty unpleasant. So that is like the next thing. But the thing is, is like, if you have a software security person, you know, they, it's just like a new Linux distro. It's a bad Linux distro, but it's like, uh, you know, it's like when they see a new Linux distro, it's like, they kind of know how to figure that out.
Right. Um, it's kinda not a hardware issue anymore, you know? So when you were, when you were kind of, uh, learning how to, uh, teach. when you were kind of figuring out how to do trainings, like how did you sort of figure out, um, what to do because, uh, having gone through one of your trainings, I can say that you are a, the way you organize information is very, uh, good.
It's, it's better than I I'm constructing the question here, but the, uh, it everything's very structured in a certain way. Like there was, there's clearly, clearly an intention behind it, you know? And, and I'm kind of wondering sort of what your theory is or how the theories of all.
Joe Fitz: So very little with what I teach in my classes. You can't just go on YouTube and find someone to teach you as well. Like it's, it's there there's no, there's no magic secret anything in all of my training. Um, but I think what I do is I curate that material. Um, so you can spend, you know, three weeks on YouTube and in that three weeks, you'll probably get like, you know, a few hours worth of good material.
That's, that's factual and current and relevant and make sense. Um, you'll re watch stuff that some stuff that's beyond your tech, your understanding when you watch it, sometimes that's remedial. Um, so what I kind of do is I try to get the, the flow from beginning to end and have it be, uh, like near guaranteed success, but not, step-by-step not spoonfed.
Um, so, and actually it builds on that even more. Um, so the idea is like, okay, if, if you've done something once, you know, what's possible, when you know it's possible, you know, how it's supposed to work, you know, and what the right path is then suddenly if things go wrong, Or if things don't work smoothly, that then, you know, when you recognize when they're not working smoothly, you can identify that you can fix it, you can move forward.
And so kind of try and start with the smallest, like step-by-step follow this procedure and get this exact output that's like foolproof. And then each lab successive, you, you build beyond that, but you get less handholding. Uh, does that make sense? Um, and I, it seems to work, um, it's kind of funny. Uh, my, my wife is a Montessori teacher and they've got a very similar concept that they use in those classes where you, you know, you explain something, you show something and then you have them do something.
Um, and it's, it's basic like anybody who's like, oh Yeah of course that's how you teach things. Um, but the reality is that, um, Yeah. Uh, it, if you stick to that, like very simple structure of organizing stuff, it comes out a lot better. Um, the other part is actually having, uh, a good idea of what the objective is at the end.
So like when I'm discussing you are, I want your end goal to be, you know, how to find pins, hook up to them, start up a console and send some commands. Right. And so once we've done that once, like let's be able to do it a second time and actually take each of those steps to do it. Um, so that, that's kinda a bit of the mindset that goes into how I sequence the information and put it in order.
Mitchell: Yeah. Uh, as someone I took your, uh, training sometime around 2018, uh, the one thing else I wanted to comment on was you all always had a very interesting selection of targets, um, which, you know, someone just sort of playing along at home, they not have a you know, set of targets that they're trying to, you know, work through it.
It's not as well pre-baked, but especially, uh, I won't, I won't give away the ending of some of the stuff that you used to do at least, but, um, were just some interesting aha moments about just saying, oh, I didn't know. That could be a pin, for example, for some other purpose feature, uh, et cetera.
Joe Fitz: I know exactly what you're talking about. Um, yeah, so it's funny because you know, when, when you're working as a pen test or you're like, you're working as product security, you don't have a choice about what you work on. Um, whereas where I'm independent and my sole purpose is to create training. I buy a lot of hardware that I take apart and it's actually fine for me to like say, oh, it's too hard to move on to the next thing.
So I get to, I get to pick out the good stuff and, uh, make use of it. Um, so, you know, I pick a wifi router that I can get reliably inexpensively and in high quantities and has all the things lined up just how I want it to. But it's still an off the shelf router. Um, actually it's for the, for the new class I've been working on I've I actually went all out and made a custom target.
That's the Best thermostat, best trademark, that model F the disclaimer on it says, Uh,
assembly required. Disassembly prohibited unauthorized. Qualified professional installation required, read all enclosed disclaimers before opening, warranty void if used, warranty also void if unused, pay no attention to the Raspberry hide behind the label. Um, so it's a, it's a little like IOT mocked, IOT thermostat with obscured chips that is like a, uh, hardware reverse engineering target. Um, and so for that class, you know, you have two targets, one target is an SSD. The other target is this a contrived, uh, thermostat. And between the two of them, you, you know, I, I give a demonstration of a process on another target.
I talk about the pieces of the process, and then we have two separate examples where you go top to bottom and repeat that whole process. Um, and so like it reinforces process and that's kind of one of the difficult things, you know, if you, if you've never done hardware and you look at a hardware problem, you're like, oh, what do I start?
But if you have a process and you've done it two times, then like, Hey, worst case, I'll just walk through the steps. It's like, you know, just show up and look pretty. And you know, everything else will be fine. Right.
Ben: Yeah. Like for, for my workshop, I basically just buy stuff at Goodwill or I find it in a dumpster and then I bring it in and hand it to someone and I'm like pop a shell on that. Bit, bit less, uh, preparation.
Joe Fitz: They annoying thing about Goodwill lately is I noticed they will part price electronics based on size. like big routers are 25 bucks and the big routers are the old ones. But the recent ones are the cheap ones. So if you want to, you know, you look at a pile of routers. There's a bunch of small ones that are like AC and N and those are five bucks.
And then there's the big, WRT 54 GS, which are 802.11 B or G. And those are 25 bucks. Just funny how they end up doing
Ben: The, but the big old, the big old ones I think are easier for beginners because, uh, th they, they actually have pin headers or, uh, populated or unpopulated. It's a lot easier to connect to a serial at least.
Max: You, do
good, good tips on, uh, connecting to Pogo pin pads? Uh, like arbitrarily.
Ben: I was going to ask that too.
Joe Fitz: So if You make your own bed of nails, it's pretty easy. Um,
Ben: It's like, how do you survive without micro positioners?
Joe Fitz: So I actually, so I have, uh, one of the, the PCB Byte is that what's called, um,
Ben: I just got one of those. It just came in the mail. Cause Matt max, uh, showed me his, uh, uh, the other day.
Max: Yeah, those are pretty great.
Ben: Yeah.
Joe Fitz: That's pretty neat. But like, I find most of the things, at least for the nature of my work, if I'm doing something, once I'm going to do it, they don't do it a hundred times. Right. And so the SensePeek is good for one off, two off. Um, if like, so for Tigard this, uh, this little orange board, um,
Ben: I'm sorry. What's it. What's it. What's a Tigard?
Joe Fitz: Tigard.
It's weird. The wonderful thing about Tigards is that... uh, no Tigard a FT2232H based breakout board.
Um, but it's got pin outs for all the, all the headers I use in my class. And then some, um, which are pretty much all the interfaces you're gonna use in, you know, 90% of your hardware hacking. So it does, you are, it does JTAG it does SWD. It does SPI, does I squared C um, and it's pretty much it works with existing software.
So like, I didn't have to write any code to turn this into a product, which is great. Um, and it basically has all the features that I wanted that no one else was making. So I finally just broke down and make it, this is, this is what happens when I don't get to travel. Right. And I, you know, actually, um, the Payroll Protection Program, You know, so that was pretty great.
Cause I got a Payroll Protection loan. I actually hired two interns over the summer. We all worked together. We designed a PCB got manufactured, put it on Crowd Supply and like sold thousands of them. So yeah.
kudos to that. So um,
Ben: What is, what is it named after? I feel like it's like a Norse myth I don't remember something or a ship. Was it a ship?
Joe Fitz: So the chip on this is made by FTDI, FTDIs global headquarters is in Glasgow and their US headquarters is in Tigard which is right down the street from Portland.
There is another called Glasgow right now, which is a, an FPGA based board based on the Lattice FPGA whose headquarters is in Hillsborough, just down the street as well.
Um, but, uh, that was basically the desire to replace everything people are using FTDIs for, which is a great mission, but it's. A little higher price and a little higher, more advanced than what I wanted. I wanted something very simple and inexpensive. So, you know, I figured I called it a Tigard because it was the next best thing to Glasgow.
So
Ben: I get it.
Vyrus: Nice.
Mitchell: Yeah, I have one, I keep it in my bag mostly just cause it's, you know, it's small enough along with the BitMagic that you can get, uh, at least at the Crowd Supply, it was coming with a pack as well, so nice little small form factor logic analyzer. And I've been using that at the hardware hacking workshops for It's, you know, uh, intuitive.
So great job on the execution.
Max: 2232 has two UARTs on it too, right?
Joe Fitz: The 2232 has two. So what I have is one of them is set up as the UART and the other one is set up as JTAG, SPI uh, I squared C. Um, and they have level shifting unidirectional level shifting. So it actually works, unlike bi-directional level shifting, which is black magic. It never works. Um, and, uh, yeah, so I
Max: I think I saw you posted that. I don't quite understand why, what bi-directional is always so bad.
Joe Fitz: Uh, do you want me to go into it?
Ben: Oh, yeah.
Max: A little bit. Yeah.
Joe Fitz: So the idea with bi-directional level shifting is you have a chip that's supposed to sense the voltage on one side and propagate it to the other.
Max: Yeah.
Joe Fitz: You're also telling that chip at the same time, it's going to take one voltage level and change it to another voltage level. It has to be aware of the possibility of taking that other voltage level and changing it back to the first one. Right. And it's supposed to do this at high speeds. Like the FTDI can go up to 30 megahertz. So it just doesn't work. It's fine for, I squared C, which is supposed to be bi-directional, it's designed to be a bi-directional protocol. It works at SPI for really slow speeds and it really doesn't work well for anything else. So, if you want it to be robust and work just, yeah.
Ben: So is the issue, is the issue something like if it gets all zeros or something from one side, it could dynamically decide that it's shifting voltage differently or something like that?
Joe Fitz: Well. So, you know, if, if both sides are zero volts, then that's easy, Right.
If one side goes up to a high voltage, right. It has to drive the other side up to a high voltage, but it has to not do that if that other side is actually being driven to a low voltage. Right. Because if the other side is being driven to a low voltage it needs to pull the first high down to a low voltage, right.
Ben: So if the low voltage side is writing all ones and the high voltage side is writing all zeroes, maybe it decides they're swapped.
Joe Fitz: Yeah. It's just, it's complicated.
Max: The conflict of, because you're basically choosing a MOSFET or something like that to do the multiple voltage switching so that has no concept of the intent of the circuit,
but it works for you the directional one way, because,
uh, you're, you're only one side is driving, so there's never a conflict. Okay.
Ben: Oh, I was just going to say TLDR black magic fuckery.
Joe Fitz: Not the Black Magic probe, a different tool entirely.
Max: Well, once MOSFETs are involved, it's always black magic fuckery.
Ben: True. Uh, and, and I mean, the neat thing about the Tigard too, is that you can also use it as a bike reflector.
Joe Fitz: It is orange. I don't know if it's reflective enough though. Well, it's funny. Um, they, I wanted to, if I like, okay, I'm going to make a bunch of these. I gotta make a bunch of custom PCBs if I'm going to do that, I should be able to lobby for like custom color. Um, and so, uh, the factory that said, like I asked, oh, can you do custom colors?
And they said, what Pantone, what do you want? Um, that was like, oh, you're the, you're the factory I should go with. So
Ben: So literally you could pick any RGB value
Joe Fitz: Pretty much. Um, and they mix it. The downside is it's a, it's a very thin soldermask, Uh, like viscous soldermask. So it doesn't tent the vias very well. If that makes any sense to anyone. When you have, when you have holes in the board and you put the solder mask over it, it's supposed to like bridge over the holes to make it nice and smooth.
But this one does not always do that. There's often a lot of little gaps, so trade-offs. I'd rather have it orange with untented vias.
Ben: Very cool. Um, so that, that's your, your first, is that your first? Uh, I know it's your first commercial hardware project. I imagine you've had a, actually, I remember you used to make little, uh, boards and hand them out at conferences for awhile.
Joe Fitz: Yeah, I haven't actually like sold any hardware though. Like, I, I like despite being a hardware person, I will, I will say there's no money in hardware.
Ben: Yeah.
Joe Fitz: And selling hardware. That is not a business I want to be in. I know people who are in it and like, I have a lot of respect for them and that's great, but like, oh, that's not my business.
Um,
Ben: It's just a matter of scale, right? Like that.
Joe Fitz: Yeah.
Ben: Yeah. Like all the, all the people making money in hardware or just like huge, uh, entities.
Max: And if you get
moderate degree of success, then China might copy you.
Ben: Yeah.
Joe Fitz: Um, people who, people who,
Ben: Like I said, it's a matter of scale.
Joe Fitz: Exactly well that's, you know. Okay. So, really, if someone went and copied Tigard right and started selling them at cheaper than I could manufacture them before, that'd be great. Cause then I just buy it from them and I wouldn't have to make them any more. Load off of my shoulders.
I wouldn't have to support them either. Um, which hasn't been a problem and not a big deal, but like, like that, that's my personal perspective on it. Um, so.
Ben: So, despite that you've got another hardware project cooking
Joe Fitz: I do. Um,
Ben: It has a name that vaguely makes me think I've forgotten about a myth. So it's, it's, it's Erebus. Right. And, and that's like, I remember, I remember Cerebus (phonetic) but Erebus, was, Erebus was a ship though, right? That was a ship.
Joe Fitz: Erebus was Cerebus's (phonetic) father, I believe.
Ben: Oh, really?
Joe Fitz: So,
Ben: Did he, did he have six heads or.
Joe Fitz: This is a PCI Express device and it's not just me. I'm working with, uh, Esden, uh, Piotr , who has a company 1BitSquared. Um, they make the Black Magic probe and the BitMagic logic analyzer. So I worked with him in the past, um, and he also helped with Tigard' but, uh, it's a PCI Express device and we wanted something that was like PCI something, but PCIE is an anagram for epic, right.
So might as well have the name
Max: All right.
Joe Fitz: And then we're like, okay, epic, what epic, what? And like, you're trying to come up with names. And so we ended up with epic Erebus, um, being the name for, for this device. And Erebus is like the primordial God of chaos or something, in Greek mythology A father to the Titans or,
Vyrus: Sounds like PCI to me.
Joe Fitz: Yeah, basically it's a hot mess.
Um, so the idea, my idea at first is I wanted this to be the small form factor as possible. Um, it's FPGA based, so we don't have to deal with limitations of a certain bridge chip that may or may not be available for a long time. Um, So it's based on the, uh, what's it called? Lattice ECP5,
Max: Oh, really.
Joe Fitz: Which when we started, was readily available and is not anymore.
Um, as soon as, as soon as we finished the board design, like as like, whoa, we gotta figure this out. And I actually went and spent thousands of dollars buying all the FPGAs, like all the socket compatible FPGAs I could. So we have enough for like prototyping over the next year. Um, because, you know, if we didn't, we're, we're stopped where we have nowhere to go.
We basically have to go and like buy dev boards and pull chips off of them just to do test and development.
Max: Go over to Hillsborough and steal truck.
Joe Fitz: Yeah. Um, so yeah, it's a, ECP5 bassed Lattice FPGA, open tool chain, and it sits on PCI Express and it's an M.2 form factor. So M.2 AE like a little wireless adapter, and we're tinkering with it to make it actually, instead of not just the release version will, may not be, may not just be M.2 AE but ABE so it'll fit in some hard drive slots as well.
So, um, and yeah, it's basically a small PCI Express device.
Max: ABE. You get only two lanes of PCI Express, right?
Joe Fitz: So AE only get two lanes of PCI Express, but what's really weird is the spec is very unclear and it's actually the only implementations of it are two separate single lane interface. Um, and that's, uh, uh, Google was Coral AI. There's a little implementation of that. So it's two PCI Express devices on one board, which is kind of weird.
Um, the spec is unclear, ambiguous as to whether you can do just one x2 device. But the thing is because it's such a small form factor you can get an adapter to put it into M.2 BM or a PCI Express or mini PCIE or anything else Thunderbolt enclosures. And yeah like, make it the smallest possible.
And then we can just plug it in anywhere,
Vyrus: And what, what goes over the PCI bus? Like what's the, what's the goal of the device?
Ben: Is how does it compare to the PCI Leech?
Joe Fitz: The goal of the device. Um, so a lot of, uh, a lot of brainstorm so far, the simplest is basically DMA attacks, write, read, and write memory over piece express because it's got an FPGA that's pretty capable. We can put a soft core on it and actually run an operating system on it. Um, or just bare metal code.
Um, we've also got a spot on it to pop in an SD card. So you can pop in a one terabyte SD micro SD card. And instead of having to tether it over USB to, you know, your attack computer, you can pop it in a system and have it dumped RAM every half hour, every hour, every day. Right. Because it's sitting there and you just go retrieve it later.
Vyrus: All right.
Ben: That's pretty cool.
Vyrus: Because it's incredibly, it's similar to something that I was going to build and ask you a bunch of questions about, cause I'm not you and you've basically built the other half of it, so. Okay.
Max: is there a, um, any other outputs inputs or outputs besides a, uh, uh, SCIO and the, uh, PCI Express interface.
Joe Fitz: got one PCB that's gonna work for two different versions of it. The first version is the basic version, which will be lower feature to be one piece of express lane. And it'll have USB 2.0, um, for exfiltrating data. So it'll be a little like connector on it that you can route outside the system to, to communicate from the other end.
Um, the other more advanced one is going to have 2x PCIE, but it actually has two PC ports, uh, two, two services, right? So where it'll be a PCI Express on one side and USB 3 on the other side. Um, but because how we've configured it, we can actually just put PC express on both sides and make this thing into a, uh, very unholy PCI Exprss bridge. Um, and theoretically, a PCI Express man in the middle with some extra hardware too.
Max: Oh, that's.
Joe Fitz: We've got, we've got feature creep
Max: flexibilities,
Joe Fitz: Um, but yeah, what was that?
Max: The flexibility of it. And,
Joe Fitz: Yeah, the Flexibility like,
Max: The, the USB 3, uh, interface reminds me of, uh, the, the, uh, riser cards that I have used at work, um, that are basically a USB 3, two USB 3 to, uh, extend the PCI Express link.
They're the cursed cables ever. But, uh, I mean the high-speed buses, uh,
Vyrus: I sense, a very interesting Bettercap module and all of our futures.
Ben: Well,
Joe Fitz: It's funny. Cause uh, you know, there, there are a couple of devices. So Slot Screamers, one that I had worked on years ago, um, and then, Ulf Frisk made PCI Leech that used Slot Screamer. And then he found some FPGA based systems that worked quite a bit better. But you know, you went from having, uh, M uh, was it uh, mini PCIE card into this huge full-size PCI Express device with like an add-on card.
And, um, um, I was excited to go back to smaller boards. Um, there is an M.2 screamer, which is, uh, another one. Uh, apparently they're very popular in, uh, gaming hacks. Um,
Max: Makes sense.
Ben: Yeah, I I've had a lot of fun with the PCI Leech.
Joe Fitz: There are people who, for some reason,wannae, uh, modify things in memory. So you can never buy these things anywhere because they're all out of stock because, you know, everybody buys them to hack and cheat and games or something like that. Systems once you've modeled them, like.
Ben: Well, I mean, that's that, then the game's over right. Then you have to get a new game system and start playing again.
Vyrus: Says says the guy, like, yeah. Okay. We're not gonna talk about that, but course you say that.
Ben: But the PCIE to PCI E bridge thing is cool. Like, yeah, you can do an interposer, um, or like sniff a device if you're evaluating an expansion board. But the first thought I had was like, what happens if you connect the PCIE buses of two different computers together?
Max: Interposers are also like, you know, $30,000. So
Ben: Oh, for sure.
Max: Actually getting, bringing that into the, uh, the the lower price range, something that sounds freaking amazing.
Joe Fitz: Yeah.
there were some, uh, very pricey, uh, law enforcement versions of these kinds of devices, um, years ago. And I'm sure there still are cheap ones are better. I mean, it's, it's gonna, it's like the board itself is expensive. We finally, we have the first prototype of the PCBs are in fabrication right now. Um, which means we still need to assemble them and we still need to actually do all the gateware which is going to be a, uh, project in and of itself.
But, uh, the hardware is that the hardware exists just need some software, whereas like software, right?
Ben: Yeah.
Max: running for a long time.
Joe Fitz: So there's a, another project Luna, which is a USB, uh, device, um, made by Great Scott Gadgets. And it's also based on the ECP5, uh, FPGA. Um, and so there's going to be some overlap in functionality and features because of, uh, the ability to go and take some of that USB stuff and just reuse it. Um, uh, I think there's some work there to get the USB 3 working natively on the Lattice FPGA, instead of having to use a separate USB 3 PHY, which be
Max: That'd be
Joe Fitz: more space, more space, more hardware, more cost
Ben: So, I guess there isn't a USB three c3n open cores then. Huh? I actually, I really love the, uh, the, the PCI Leech and, uh, I've used it a lot over the years. Uh, it's really good for like BitLocker bypasses. Um, as soon as all put the link up, uh, you know, talking about the project, like in the early days, I just ordered one of those chips, like the 8898 or whatever it is, you call it the Slot Screamer.
Joe Fitz: USB 3380.
Ben: Yeah. 3380. Yeah. I have number dyslexia. So the, um, I just ordered one of them. And so, and then I got the Thunderbolt adapter and then like express card adapter and all that stuff. Uh, and I was like set up to do DMA attacks. Um, and you can bypass BitLocker. You can like log into a locked to, uh, you know, uh, computer.
Um, and what I was asking was, uh, well, it's really, it's actually, it's really exciting that you're making this board because, um, people haven't been able to get the PCI Leech hardware, uh, for a long time. And, uh, this would sort of, um, you know, when people are like, ah, I want to get one of those. I can just be like, go buy Joe's board.
You know, you don't have to borrow mine. Um, but, uh, I was wondering, um, if you'd considered like porting the PCI Leech software to your. Uh, whatever the operating system is, you put on there. Um, and cause at, at the core, it's basically just the ability to scan. Like you need the trampoline to access high memory, and then it's the ability to scan for kernel, signatures, and memory, and then like patch things at an offset.
Right.
Joe Fitz: So, so you're basically saying, well, can we, can we put, uh, put the logic into the device itself and yes, absolutely. Um, so I think there are going to be a couple of different use cases though. I think there's going to be that interactive use case that a lot of people are familiar with. There's going to be a forensic use case, which is going to be full speed, memory dumps.
Um, one thing that I've noticed about that, though, is you, okay, so you've got four gigs of memory you want to dump it. And so you want as many PCI Express lanes as possible, and you want as fast USB as possible to get that off. Well, nobody's compressing that, right? They're just grabbing bites and shoveling over those protocols.
Right. If you compress it suddenly, you know? Yeah. You're your bottom line is still going to get a PCI Express feed, but you can cram memory down a lot if you want it to, to, to dump it off. Um, nothing's doing like inline diffing. So if you know, you take a dump once and you pass it off and then you like get a diff of like what's changed since your last dump.
Um, that should be a lot quicker over on a narrow channel. So that USB 2, you know, that the cheap, you know, Erebus basic USB 2 version should still be able to do some pretty. Uh, fast memory dumps with all that mine, all about taking in consideration because you got a capable, uh, chip on the board itself.
Ben: Holy shit. So you're talking about doing like cheat engine stuff on like the whole computer's memory.
Joe Fitz: Well, I mean, uh, I'm talking about it now cause I haven't implemented it yet and then it should be possible.
Max: Uh, since, uh, um, it's FPGA you, you may, I mean, depends on the PHY you end up implementing, but you may be able to bypass having to have a high memory loader as well. you might
Joe Fitz: Oh yeah. So it's going to be 64 bit It's going to be 64
Max: Oh, a hundred percent
Joe Fitz: Yeah. Yeah. Um, now it's going to be, what's going to be more of an issue is how apple and now Microsoft finally have implemented VT-d for protecting against malicious, uh, PCI Express devices. Um, the solution to that is you're going to have to be a little more malicious about how you pretend to be a device.
Right. Um, so you know, the initial bypass of MacOS's Thunderbolt productions was to pretend to the, a Broadcom SD card adapter because of Broadcom SD card adapter was on the OK list to get its, uh, guard dropped. Um, so you just gotta figure out what is on the OK list and just be that way.
Ben: Well,
Joe Fitz: know,
Max: Or you could potentially write to another memory space, uh, uh, controlling one of the other
devices that's already on them.
Joe Fitz: Again, if, if that is enabled on that system,
Ben: I mean, as I recall though, everyone was super worried about VTD, but, uh, as I recall, like all VTD lets you do is mark, which devices are internal and which devices are external. Right? So.
Joe Fitz: you get to throw devices into like a separate memory space. So it's like throwing, uh, you know, uh, uh, P page memory, uh, uh, address mapped for page walking for memory for device,
Max: You
can, create as many groups as there are ACS groups on the, on the system. It's not just two, two slots, right?
Joe Fitz: Yeah. Yeah.
As many as you want. Um, all right. As many as the bias has configured a system to support properly.
Max: Yeah.
Ben: So, I guess the, the attack from the outside is you have to look like a device that has some kind of legacy legacy properties, uh, or you have to connect to an inside port that is not as locked down.
Max: You have to have your ability to USPI ISA bridge.
Vyrus: It's actually, this actually really reminds me of like, you remember bah, ah, man, it was like the early two thousands ish where like the, like the popular thing for HP and Dell laptops to do was to start white listing the like hardware ideas of like mini PCIE wifi cards. So you'd like go to update your, like you buy a wifi, you buy a laptop that had like a janky Broadcom in it.
You'd go to put like an, a throws chip in it. Cause you wanted to do some like cool wardriving stuff and all of a sudden, like your machine wouldn't boot and your bios would just tell you, oh, thing bad. And you'd have to like bang your bios to like change the, the Mac address to like be the right number.
And then it would work like this sounds like the same thing, which is silly.
Joe Fitz: and, and then also like getting into low end servers and disabling the use of like a standard graphics cards, because they don't want you to just buy the low end servers. They want you to buy the high end overprice gaming platform. Right.
Vyrus: Oh my God. It's FTI all over again.
Joe Fitz: Uh, but yeah, um, you know, you, you got to, you're going to have to get crafty at some point once operating systems start locking stuff down. But whether that means they're going to be a, uh, a bigger demand for the men in the middle functionality, um, or whether you're gonna have to do tricky stuff, like putting your computer to sleep and swapping your network card out and putting in this card, because you're when it Ricard is trusted.
Um, like stuff like that.
Ben: I see. So the, the man in the middle of functionality for PCIE would be the killer for that. Cause it's not like they're doing a kind of a Diffie-Hellman key exchange
Joe Fitz: Yeah. There've been,
Ben: sorry, go ahead.
Joe Fitz: there've been proposals for authenticating devices, but I don't see any of them as getting implemented anytime soon.
Mitchell: Right.
Ben: No,
Max: The
Ben: that it requires buy-in from every, uh, hardware manufacturer basically. Um, yeah,
Max: mean that that's what PCI express is buying from all the hardware manufacturers, but
Ben: yeah, but I mean, that'll give you at least 10 years, right?
Joe Fitz: but, but, you know, P piece of express is buy in from everyone because it is like the best engineered interface that makes it as easy as possible to get interoperable without effort. Right. Whereas any security feature is the exact opposite.
Max: Yeah.
Joe Fitz: I like here, we want to add more features to prevent interoperability and give you reasons why customers are going to call up and complain that their system doesn't work, which is like,
Max: It's a surprisingly robust protocol too. I remember seeing some of these
together. 13 riser cards like
over two meters long.
Joe Fitz: Nope.
Ben: So if you could man, in the middle of P so if you can, man, in the middle of PCI express, you basically just wait until, uh, you see some other like valid DMA message go by. You can, uh, you know, pick out whatever identifier you need from that, and then just generate your own with the same identifier, right?
Joe Fitz: you would sit in between the network card and the PC. Right. And you just let everything pass through. And then when you're ready, you just pretend to be the net Ricard and do a DMA request to something that you want. And don't pass to the note card. Um, there's a little bit of, of messing you have to do with, with, uh, uh, packet ID numbers or whatever, but, uh, yeah.
Max: enable the, uh, some bus spins as well. there's, uh, some fun, fun, fun stuff with BMCs in, uh,
Joe Fitz: Yeah.
Max: is
Joe Fitz: Well, I mean like the piece of express spec has jaytag pins on the header as well. Um,
Max: Uh, but those are for manufacturer, right?
Joe Fitz: you know, maybe it's it's it's, it's not well-defined, you know, I, you wouldn't put it past a beginner board designer saying, Oh, this is JJ here. I have a JAG there. Let me wire him up and, uh, accidentally giving jaytag access to every PC express device.
Like, I mean, that's, I've seen how boards are made. I've seen,
Max: I mean, like. You know, for some purposes that may be good. Like it's a common interface, you know, be easy to do more tests that way,
Ben: like that all sounds like useful stuff, but I'm still, I'm still wondering, like what would happen if I just, you know, use that as a PCIE bridge and just connected two computers together.
Joe Fitz: so, um, there is a feature called non-transparent bridge, um, which basically does that. Um, and you basically have a memory window that you can pass stop between two systems. Um, it's like a cheap, uh, interconnect.
Vyrus: I didn't Google didn't Google like do this for awhile.
Joe Fitz: Intel sports it on a bunch of processors and
Max: yeah,
the
Joe Fitz: it exists.
Max: there's a multi-room, uh, there, so there's Sri AOV, which is like a, I over chosen for a single complex, but then there's also multi-room PCIE complexes and multi-room I overgeneralization just so you know, add turtles together.
Vyrus: Well, as soon as you remember, like, this is how, this is how Google decided to not roll their own hardware, because they basically did and they got it to work. And then it was just too expensive to be their own hardware company and be Google. So they stopped doing it, but like, that's what they did. They had PCI, like they stopped using networking.
But aside from all this other stuff that they did, where they had like custom boards and custom chips and custom everything, like they literally stopped using networking cards. They just had PCI to fiber that like the other box was just an address on the PCI bus. And then that was what drove them to drive these giant management systems that are only now starting to get open source, like.
Ben: Holy smokes. So what do you even do with it? Just let you put like, like 16,000 GPU's and one cracking rig or what, like
Vyrus: Well, it
Ben: are they doing with that?
Vyrus: well, I mean, I don't know what they're doing with it now, but what they were doing with it is it basically gave them the equivalent of like open MP style supercomputing without all the overhead and management of open MP. Like you could just do stuff and instead of basically you'd use code for interprocess communication and interprocess communication, the other process might be on another machine, right?
Joe Fitz: But let's look at this computer. Supercomputer magic is not just the CPU, but also the interconnect. And it was just like a, Hey, we've got commodity PCI express, which is fast, cheap, and low latency. And just use that as the interconnect.
Max: Yup.
Ben: So you could make like an Airbus, a supercomputer.
Vyrus: Yeah. I, I,
Ben: Hmm.
Vyrus: like I'm thinking about like, so like if you like the phones run my lightening now, right. Or so like, if you could run PCI over lightning. So if I had a bunch of phones that were just PCI in points and I wanted to fund the thing
Joe Fitz: uh, I phones, iPhones have NBME storage and Vme is PC express. Right.
Vyrus: oh,
Joe Fitz: Um, you know, my, I have a camera with the Xpress card storage express card is PCI express, right? The Xbox and the place to, I dunno, PlayStation Xbox, it uses basically a express card pin out compatible devices, basically NBME memory cards,
Vyrus: This is where I feel like the market for like old I-phones with broken screens that no one wants to fix, that you could just in a giant PCI IRA and like, oh, now I can fuss iOS programs. How that happened.
Ben: well, well, well, so express card is either PCI express or USB, uh, depending on, um, uh, basically it has all of the pins of USB and then all of the pins of PCI E and then there's a, a selector pin, which, which, which says, which one of the, which one they've implemented. And, and, uh, every express card thing I've seen is either one or the other, but I bet you, the storage ones are using.
Joe Fitz: um,
Max: So does that like USB
Joe Fitz: uh, CF Xpress, CF express, uh, compact flash express. So compact flash was the old, you know, memory card standard and smart media and compact flash where the contenders compact flash was like basically the pin out of card bus. You know, in a different form factor, which is, very similar idea, but, uh, then they had a, a sadder version and now they have a piece of express version and that's what Nikon camera uses.
And that's what the X-Box the new X-Box has. So, you know, I've got a picture somewhere where I got the Xbox memory card and popped it into my camera and it didn't work and identified the device. It just, yeah,
Ben: and the, and those cards are using the PCIE circuitry, not USB.
Joe Fitz: they're using, you're using piece of express. Yeah.
Ben: Wow.
Joe Fitz: Yep. That's how they get such a fast, uh, interface. So
Max: It's it it's ridiculously standard these days. So
Ben: And in the future, is everything going to be PCI express
Joe Fitz: I don't see why not.
Ben: weird?
Joe Fitz: I mean, it it's, it's fast. It's cheap. It works. It's proven. Like, is there ever is when is the next time, like, when is there ever going to be a case where it makes sense to design something from scratch when you could just use PSX breasts?
Max: yeah, it's just basically memory mapping as a protocol.
So,
Ben: Well, they'd better be really confident that VTD is a sufficient security. Mechanism
Max: well, it's, I mean, the, the, the fight thing is a piece of express is just a networking protocol, like, uh, at its
It's, not even really a bus protocol. It's like, it's. So it's more, it's like, it's more like, uh, uh, more like IP or personally.
Ben: that's deeply trippy. Uh, no. So PCI express, um, uh, I, I was just sort of thinking about the physical connections for PCI express. There's like a, it can go over lightning Thunderbolt, USB three,
Joe Fitz: Well USBC, but, you know yeah. And USB 400 ball. Yeah. Uh, there is actually a specification for external cabling, but it was never actually really used. So,
Ben: Interesting. So if you, if you had the Arab is, uh, set up as a, uh, a bridge, what would you do for cabling?
Joe Fitz: so if it were in a bridge and I still was like in an attack mode, right. What it would be as a connector in the target system. Right. And then there'd be a flat flex cable. It would come out the back with another PCI lane. And that would go into like a dummy car that would adapt it into a piece of express, which would go into a Thunderbolt and to my laptop or into my laptop directly or something like that.
Um,
Ben: So just a flat cable.
Joe Fitz: flat flex cable. Yeah.
Ben: Awesome. Wow. That's super wild.
Joe Fitz: Yeah. I just got, I'm seeing that the board's going to get shipped sometime next week and then we'll get it and then we'll have to work on it, which.
Ben: So how many, uh, how many revisions of aboard do you usually go through? And I'm asking, I'm asking for a friend, a friend who has just gone through four revisions of a board with each, with their own unique, stupid errors on them.
Joe Fitz: it depends. Um, so for tiger, we had the first revision was Tigered, was, was, there was a prior project, um, uh, time P, which was, uh, basically the same design. I had talked to the guy who put together a David Thomas show and, uh, he, um, had laid it out and so photography, we want to something from scratch with a few different features, but it was the same concept.
So like we kind of had a pre prototype already and it's a pretty well understood chip, very widely used. And so we still had our revision one and our version zero, which was our like test platform, which pretty much mostly worked. And then we had, um, the, you know, almost production ready 0.9, um, which we actually made a small volume run and still found some issues with.
And then we had 1.0, which worked great. And that's what sold, we sold with crowd supply. But I still noticed that there were some issues with the spacing of the connectors, where if you had two, uh, flat connectors connected at the same time next to each other, there wasn't enough room. So we made a minor adjustment for version 1.1, where we just put a little more space between the connectors.
So four, four to five water revisions for a simple board that even worked the first time. I
Ben: Oh, good. All right.
Joe Fitz: don't have no idea how many board revisions are going to be for a board that is dealing with five gigahertz signals. And, um, yeah. So it's got an FPGA. It's got, it's got a 286 megabytes of Ram.
It's got eight gigabytes of flash. It's got USB two and three. It's got four power, uh, regulators, um, for all these regulators. So it's, it's uh, it's not a simple board.
Ben: So what I'm, what I'm hearing is I'm still in the normal range of normal expected failure. I mean, my friend is in the normal range of normal expected failure. Yeah. So nothing to be ashamed of. That's good to know.
Joe Fitz: no,
Ben: Um,
Joe Fitz: to me, assuming you have something that works, right?
Ben: oh yeah. Well, it's, it's um, it's, uh, I I've just been, uh, I've been making a thing that runs on 18, 6 50 batteries. Um, and so when I screw up, I basically just made a really complicated Firestone. So it's.
Vyrus: or vape
Ben: Yeah, but it's, it's not vaporizing anything you want to inhale. That's the problem.
Vyrus: doesn't mean it's not a vape.
Ben: Yeah. It's an Abrams something
Joe Fitz: Yeah.
Ben: for sure. Um, awesome. So let me ask you, um, uh, you're, you're, you're way farther down the line of like designing and, um, and, and building boards. And I am actually you, uh, I just learned a key CAD earlier, uh, earlier this year or late last year. Um, and you, you gave me probably the most useful piece of advice for learning key CAD, which is that, uh, what did, what did you say?
You, you have to, uh, pretend it's like a mouse with 101 buttons. It's like, you just, you never click anything. It's just, the keyboard becomes the mouse
Joe Fitz: over and know which GM keyboards press.
Ben: Yes. Which it's another interface designed by someone that hates people, for sure. Um, but once, once you
Max: think mostly Germans.
Ben: th I, I didn't want to get racial with it, but I I'm pretty sure that open OCD and key cab were both designed by Germans.
Vyrus: Jeremy does not a race,
Ben: Well, w what, what is it? I don't know. The species subspecies and nationality. Okay. Um, I, I don't know. I just got into someone, got really, someone got really mad at me for saying that. Yeah, grizzly bears and brown bears were the same species the other days. So now I I'm afraid. I don't know what words mean at this point.
Vyrus: Um, yeah, I can echo that sentiment. Like, I don't know Jack about Jack when it comes to hardware, but like I managed to learn. KiCad like, I, now I know how to make all the lines and the parts and things. And I know how to like import symbols from random places on the internet. Like,
Ben: th th th the thing about open.
Max: wait until the next major version where they change all of the key Kim
Ben: Oh, no, I'm not, I'm not, I version five forever.
Vyrus: yeah.
Ben: I'm not tracking with version six.
Vyrus: Isn't that where Harvard people do. I mean like the one red team I did where they had like electronic microscopes, they were all running on XP.
Ben: Yeah. Well, that's good because
Joe Fitz: taking place machine runs XP.
Vyrus: See.
Ben: I was going to ask you about the pick and place. Uh, so I, I, I, I'm kind of, I just got my own reflow oven, like about a year ago. Uh, so now I can like order the board from OSH park. Um, you know, I can cut a stencil on my craft cutter. I like, you know, put the solder paste on and then I have to like tweeze all the components on the second in the oven.
Right. So I'm kind of up to that part and I'm pretty comfortable with that. Um, but, uh, I wanted to ask you, if you, do you have like a personal use pick and place like in your house?
Joe Fitz: I arrived today.
Ben: Oh, goodness. So you've just crossed, you've just ascended to the next level.
Joe Fitz: Just, yeah, I've just, I inherited a new project.
Ben: Well, so strangely enough, I've been kind of lurking around the idea. So I've been on this mailing list of people that bought this Chinese pick and place machine from like two, three years ago. Um, and I've just, I follow that. That's the one mailing list where I follow the traffic. Cause I'm waiting until it seems like it's safe to go in the water, you know?
Um, but it it's become like a, like a part to full-time job for most of them just trying to figure out, uh, how to, how to it's kinda like a Audrey too, you know, like they just keep giving, feeding it more and more blood.
Vyrus: So do we have an over under on when China launches an Aurora based, just have American hackers because like they all pick and place their own stuff now. And it's way easier to get for wearing that way.
Ben: Well, the first, I mean the first, the first thing you do when you get, when you get some equipment from China is you like put an alternate firmware on, that's usually written by someone in Germany, right. So at least it's spread around a little bit in terms of the supply side risk
Joe Fitz: Yeah,
Vyrus: you know, Israeli,
Ben: or yeah. Or wherever, you know, whatever, whatever, um, nationality
Vyrus: you swap one group. If you swap one group of spies for another and pray that they don't talk to each other.
Ben: You know?
Joe Fitz: want them all on there so they can watch each other.
Max: Yeah.
Ben: Yeah.
Joe Fitz: not going to use their good stuff if they know each other's watching,
Ben: That seems, that seems sound so. Okay. So you just gotta pick and place machine today. W w which one did you get?
Joe Fitz: I got a Neo done for
Ben: Oh, where's that? Where's that from?
Joe Fitz: um, but it's, it's kind of a step above the charm high tier of desktop ones,
Ben: Yeah, the charm. That's the mailing list. I'm on that? It seems like that one might be, might require a little too much handholding for me.
Joe Fitz: Yeah. And I was, I've been eyeing those for a long time. I've probably been on that mailing list looking as long as you have, um, I've never actually needed. I still don't need one. Um, that's the funny thing. Um, but it, um,
Ben: I mean, w what's need got to do with anything.
Joe Fitz: exactly. Um, but yeah, th the Neogen was a smallest, the biggest, the highest end one that would still fit in my office.
I'm still is still kind of like a desktop ish size. Um, it's just, it's got foreheads and a few more features, a little more advanced, uh, theaters than, than the charm high does. Um, and the reason I've got it is because I was procrastinating trying to finish the applied physical text to class and surfing the web and not doing the work I should be doing.
Um, and that's kinda how all my projects start, but like, I do a lot of like one off or 10 off things. And I always avoid, um, small components because I just I'm, I'm so impatient when it comes to assembling them. Like I, um, I use a lot of modules, um, and so like, I've got a module with a microcontroller and a bunch of pin outs and everything, and just cause it works.
Um, but when I scale that a little bit beyond that, like, I'm not sure I'm ready to go and talk to a contract manufacturer to have it done. I definitely am not going to do it myself. Um, I'd really love to have something that just places, all my passive components for me. Right. So if I have this thing loaded up with all the standard resistors and capacitors I use, and when I design a.
I just have it, do all the resistors capacitors and not just do the one or two other chips by myself. Like that's going to save me time. Um, because it's yeah. Now the dilemma is right now, you can get a JLC, PCB and PCB way. They'll also do like manufacturing, like that will they'll manufacture your PCB and they'll put a bunch of components on it, really cheap if you use their standard life part library.
So I guess I'm kind of implementing that too, but maybe I'm just hedging my bet for like the, the, the blockade that's coming across the Pacific ocean that, you know, we're not gonna get stuff anymore, or I'm not going to get my DHL packages from China in two days. It's going to be, you know, a more realistic, you know, week or so.
Ben: Well, if that happens, I think electronics in this country is going to be basically over. Cause where would we even get semiconductor?
Joe Fitz: That's why I've got a huge box of them,
Ben: Oh, you're just stockpile them.
Joe Fitz: Yeah,
Vyrus: I'm the south American semiconductor farm began.
Ben: It's in there.
Max: yeah, we're going to have to have
Ben: with the pure one toilet paper.
Max: we're
them off of a old, a badges.
Joe Fitz: Yeah.
Vyrus: that's
Ben: I hear badges.
Vyrus: maybe that's,
Max: Yes.
Ben: Well,
Vyrus: we're going to revitalize the energy sector. They're all going to make semiconductors now,
Ben: oh man. So I do this thing where I
Joe Fitz: When you got reels,
Ben: yeah, right on.
Vyrus: as he holds up a box of real.
Ben: Yeah. I kind of decided like when I started designing boards, uh, I just decided the smallest thing I can comfortably tweeze, you know, onto a board is like a 0 8, 0 5 size, you know, resistor. So I just designed everything around 0, 0 5. And I was like, oh, that's great.
And like, anything will be bigger than that. But I kind of ran into a problem with power reg like switching power regulators. So like the buck bust, literally got buck converters, boost, converters, buck boost, you know, like a lot of those no longer come in a size that is like manageable by a human. And they, they don't have like a bigger package available.
Um,
Joe Fitz: yeah.
Ben: and, uh, and that's kinda where I started thinking maybe I should get a robot, uh, because, uh, man, those it's like a. It's less than two millimeters, two millimeters long, and then a little narrower than that. And it's like ball grid array. Um, and what makes it worse is it's like two by three pins. So there's a right orientation and a wrong orientation.
And you can't actually read what's on the top of it without a microscope. Uh, and, oh man. So if you drop it and like flip it, you have to like, like get the microscope out to see which way it landed. It's just a nightmare.
Joe Fitz: that's, that's really part of why I don't actually need a pick and place machine because like, I, I kind of standardized it on a 600 threes because I did it a few years ago when I could still see them. Um, uh, I can't really anymore. Um, but you know, uh, charm high, the annex, the inexpensive desktop. Like it can technically do a 600 threes that advertises
Um, but it doesn't do a great, um, so it's that tier of machine is not going to be able to do anything I can't do by hand. It's just going to do it quicker. Um,
Ben: And that's, that's kinda what I, what I decided to cause all the people on the, on the list are complaining about how it is not putting things down correctly and needs to be recalibrated all the time. And I'm like, I don't need a robot that can do things worse than me, but that sort of defeats the purpose of the robot, right?
Joe Fitz: I don't need a job as a babysitter.
Ben: Yeah.
Max: Yeah.
Joe Fitz: I'd rather, I'd rather, I'd rather use tweezers than sit there as a babysitter.
Ben: Or a robo sitter.
Joe Fitz: Yeah.
Vyrus: I just like the idea of picking parts for comfort seems silly to me after like, like the third thing, maybe I learned how to surface a mountain solder was approximately three. I just gave up comfort.
Joe Fitz: Yeah.
Ben: so, uh, so what do you, what, what are you going to, uh, what's the, what's going to be the first victim of your picket place machine.
Joe Fitz: I have no idea my
Ben: Oh,
Max: Your time. Yeah.
Joe Fitz: my time.
Vyrus: I mean, obviously he's going to build a pick and place machine, right? That's how this works. It's like 3d printers.
Ben: going to get so angry at this pick and place machine. He's going to make another one.
Vyrus: It's like, it's like 3d printers, right? You get a 3d printer and then you print a better 3d printer.
Ben: Wait, I, I got dibs though. Like I, I have a different project idea. So you were doing like a PCIE bus enter poser. Um, you know, it would be great as a lower cost Saida bus. Interposer
Joe Fitz: Yeah, the
Ben: that out there. Yeah.
Joe Fitz: uh, well, the dilemma there is what's the trip you put inside, right?
Max: That's
the best I've ever say to that.
Vyrus: I suck. So it's all of them, right.
Joe Fitz: I think Travis or surgery brattice came up with that one.
Max: Okay.
Joe Fitz: Um, because I know Travis looks look years ago for like the, the SATA controller to, to make the, the Santa face dancer, um, and was unsuccessful in finding anything that would, would work for that. Um, all the SATA controllers like that you get on drives are, um, like unlisted on purchasable parts. Like, so you look, you look at the ships on the drive, Samsung, Marvell, everyone else.
Like they make them, they don't sell them. They don't list them. They don't have data sheets, nothing. Um, you know, you can find something on open corners and use an FPJ, but then again, you've got another project in addition to, you know, your pick and place machine and your PCB mill and your, uh, open cores, you know, Verilog implementation of Satta.
Max: Has anyone
Joe Fitz: haven't even gotten to the problem you want to do, which is actually to have a, uh, a sediment in the middle device, right?
Ben: Yeah. Well, there, there is one, there is, there is one
Max: anything off of open cores that worked by the way?
Ben: I haven't. Um, but I heard one time someone took an eight 50, an 80, 51 core off of there and actually use it for something, but I don't remember any details.
Vyrus: Yeah.
Joe Fitz: open. I've got open risk running, uh, before on a, on a digital and board. So,
Ben: Um, yeah. So for, for Satta, a man in the middle stuff, um, uh, there's actually a lot of, kind of vulnerabilities there, uh, that, um, you can kind of get to, if you manage to like reverse a disk controller, firmware, um, But, uh, it would be, you know, just like it's easier to, you know, spot, um, web bugs from a man in the middle position.
It's easier to spot a sate of bugs from a man in the middle position. Cause usually there's like proprietary vendor commands to update the firmware. Right. And that was, that would be kind of how you would implement an attack is like, oh, here's some malware that like then goes and writes itself into your hard drive controller, firmware.
Um, so good luck with all that. Um, uh, cause kind of the neat thing about that is up until recently there was absolutely no way to verify the firmware that was in a hard disk controller, like from the CPU. Um, I, uh, I heard a rumor that Intel is like threatening to come out with some new spec for like verifiable disks, uh, or just controllers.
But um, I don't know if that's hit the market. I don't know if that's going to be a real thing. Um, but uh, no, I, well, I used to complain about this a lot, uh, at a previous job. Uh, because they're actually have been weaponized, uh, uh, disc controller root kits, uh, in the past. And that lets you do fun stuff.
Like it lets you work around some types of full disk encryption. Although not the ones that don't actually require the hard drive to do anything, which is most of the modern ones. Uh, but it, it, uh, on un-encrypted drives it lets you do fun stuff. Like anytime somebody like reads a binary from the disc, you could like insert malware in it.
Um, and obviously, uh, disc malware can hide as much, uh, info as it wants because it's got the whole hard drive. Um, uh, and it can control like how much of the hard drive you can actually see. Um, I don't know. There's a lot of neat tricks you can do. Um, but they're actually, there have been incidences of, uh, of hard disk malware before.
Um, and uh, the, the key thing is there's these like proprietary vendor commands update the firmware. Um, and they're usually pretty difficult to track down. Uh, but if you had even just a bus sniffer, right? Like not an interposer, um, you could just run the, uh, the vendors, a disk update command and be like, oh, there's this ADA command that updates the firmware.
Like, we'll go with that. And, um, also
Vyrus: how Western digital found the China ones. Right? The Chinese microcontrollers that were on the hard drives and it was like a whole bunch of stuff had to get through.
Ben: Yep. And like historically, uh, hard disks also, haven't done a great job of requiring like cryptographic signatures on their firmware updates. Um, you know, the usual usual stuff. Um, the other thing about disc controllers is it's a very narrow market. There's only like three suppliers. Um, so there's a, it's, it's surprising and they're all arm based.
So it's actually like surprisingly a homogenous,
Joe Fitz: yeah, until we even had armed just controllers,
Ben: you go, Hey, into Intel made arms and those voting machines had Intel arms in them.
Vyrus: And NVIDIA's making Nvidia is going to make arms now, supposedly as a general purpose CPU,
Joe Fitz: we're just surrounded by arms dealers.
Ben: Uh,
Vyrus: every sense
Ben: Y
Max: I
say.
Joe Fitz: that our joke for the, for the episode?
Ben: Uh, well, I think we're above our quota, so, oh, you mentioned your new class. I forgot to ask you about this earlier. So, um, so you were doing all these, like, uh, just kind of if we can get off our Arabic. Yeah. Yeah. We'll go back to the training. Um, so you, you were super busy doing training for years and years.
Um, and you know, traveling all over the place. You've been at black hat. How many years in a row? like, like, I dunno, eight something,
Joe Fitz: something like that. Nah,
Ben: 6, Somewhere like that. Um, and, uh, uh, so when the, when the, when the pandemic started, you're like, oh, Hey, suddenly I'm not traveling so much. Um, how did you kind of like pivot to, so you're doing like online stuff now.
Right?
Joe Fitz: So, um,
Ben: How does that
Joe Fitz: it hit, it's interesting. Uh, so when, when it hit, um, kinda saw the writing on the wall and didn't expect black hat to be happening that year. And so like even before black hat made their plans, I'm like a heads up. I'm not doing four separate hardware classes this year. Like even if everything gets better by them, like I'm getting ready to do what I can do remote.
So like, it was April when I was already thinking about how to do anything were about, so I, I put off classes two, three, and four and focused on number one, which is one I've done long enough and repeatedly enough that like I could probably actually make it happen. Um, and so part of that was actually kinda comprehending what it meant to actually do a remote class.
So when I'm doing a live class, people have a dedicated time to do it. And even when I go onsite to companies, like it's funny as much as they're paying for training, what they're really paying for is the like, uh, corporate culture, acceptable excuse from checking their email constantly for a day or two or three or four.
So like, that's what, that's what they get. They get to focus on something for a day because training is okay. And so I go and I show up and I have built a whole class around that Yeah. Keeping attention for that, that eight hour period, multiple days in a row. But when we switched to remote, like it's not going like that.
Um, you know, there are classes who will try and like do it like that, but you end up with the worst of both worlds. You end up with the, the fact that you have to be in the same spot for eight hours of a real class, as well as the, like, not understanding what's going on and miscommunication of a remote class.
So I basically went straight for the self-paced version. Right. Um, I took all the class, the labs that I have, I used to try and do like a half hour lecture, 90 minute lab time in my life classes. And I cut that down. So it's five to 10 minutes of lecture and then 15 to 20 minutes of lap time. Right. So making it a smaller chunks, because the reality is people are distracted.
They're at home, they're at work?
they're, you know, picking up with kids, uh, all sorts of things. Um, and so, you know,
Vyrus: Speaking from experience, right.
Joe Fitz: yeah, no, no, no, no. It's, that's like, um, but yeah, being able to go and work on something and learn something in half an hour is pretty much what needs to happen if you're going to have a sustainable self-paced class.
So some people will do the class and they'll do half an hour a day. Some people will do like an hour a week. Um, and it doesn't matter because the sequence of the class is all there. Um, the other thing that benefits is. No. When you, when you teach something, you have all these tangents you want to go down and talk about, and I could, I could talk for three hours about you art, and you know, you aren't Pinots.
Um, but no, one's going to want to hear that, but I can go and I can do my five minute lecture on UART pin outs and then do three side lectures on a Sotera topics about UART peanuts and link them down at the bottom and people can go and watch them and get that supporting information, but not cloud up the ones that are just having a hard time focusing on the core, core function, core, core bits.
So that's kind of the difference between a, an in-person class and a remote class, snipping it into those half-hour chunks with no more than five or 10 minutes of lecture at a time. And like hands-on where you actually get something done within a short amount of time. Um, so that's worked, uh, I did black hat in 2020.
That was the first time I did a, remote class. And then after that, I opened it up to like individuals to go to my website and just register. And I, I shipped them a box of hardware, all the tools they need, and they can go and walk through the class, uh, with video lectures, followed by labs, followed by video tips and hints followed by walk-throughs, um, which has worked pretty well for lots of people.
Um, and so kind of let that roll and then it was very clear that like, okay, 20, 21 is probably not going to be all that much better than 2020. So maybe I should think about doing another class. So I, I had been doing a couple of. Remote private trainings, where they wanted the four days of training. So I had already done this class remotely.
I wanted to make that leap remote self-paced as well. So my goal was to have it done August 1st, then September 1st, then October 1st and then November 1st and then November 2nd and now never number third. And so like, I finally have it up on my website. You can go and register for class number two. Um, but yeah, it's been, uh, it's, it's amazing how much work it takes to put something like that together, knowing that it's going to be viewed many, many, many times over and over again,
Ben: a, w w were you doing remote training before the pandemic? Like once in a while.
Joe Fitz: never, I wouldn't even really consider it.
Ben: So it was this all just like a theory. You, you came up with that worked out or, or, or like, how did you come by this? A half hour increment, uh, number, cause it, it seems actually really insightful.
Joe Fitz: so when you go and try and look for information about online learning, you dive into this, Um, cesspool of sales and marketing, and you know, you, you, if you ever heard of a sales funnel, right? Well,
Ben: like the Sandler.
Joe Fitz: we've got sales funnels of sales funnels, Inc. Leading you down the path to buy a sales funnel. So you can have your customers go down your sales funnel and funnel up your sales all over your funnel. Right? And it's really disappointing because like I actually like, so there is there sales funnels and then there's corporate mandatory training, right? That's remote training, as I could find really open resources about it. And I don't want to do a sales funnel. Right. And I don't want to do crappy corporate training.
Right. The whole thing about crappy corporate training is like, it's, it's mentory compliance training. You need to go and make a training. That's just enough to keep people's attention for long enough that they can check a box in the right spot and get credit for doing like whatever and certifying that they're not gonna touch, you know, uh, bloodborne pathogens or they're not going to like touch other things they shouldn't, or, you know, whatever, like all these things that people apparently need training seldom to do that, you know, uh, I don't know, compliance training.
Um, but if you look at a lot of the objectives that you put in, like when I tell you how to make your mandatory, you know, Sarbanes, Oxley, like compliance training, like they give you all these tips, like do this, put images in it, you know, chunks of time. And all those tips actually are relevant to actually teaching useful information. Uh, because they're tips, they're tips about keeping people's interest and they're like, and you watch TV, right. And movies. And like, you watch a movie from the eighties and nineties, are you watching movies from today? And it's really different how they're paced. Right. And they're paced do to just get people's attention.
Um, And I didn't think that I could go any shorter than half an hour and a five minute lecture to get enough useful information. So, that's where I settled on that. I settled on it because it was the smallest amount of time I could get, um, useful information in a lecture and useful compliment minute in a lab.
Um, and that's where it came from. That makes sense.
Ben: So, I mean, if there's some, some thankless dude somewhere, uh, like maybe did some research or whatever, and then like, It had the, the horrible job of writing up like helpful tips to make your Sarbanes-Oxley training more useful or more interesting. I like to hold people's attention and they actually did their homework at some point.
And then wrote that down when the truth of the matter is you might as well just give the fuck up for Sarbanes Oxley training, but somewhere, somewhere did a good job and you will have found their notes and like repurposed it to, to the actual, like, useful effect. That's got a nice,
Max: well, it reminds me is, uh, there's this Pomodoro method of, uh, uh, sort of a task management where it's like you said, like, uh, one of those tomato clocks for 20 minutes, and then you like work for 20 minutes, then take a small break, uh, like attention break to something else, you know, check emails or something like that for another 10 minutes.
Or, you know, even if it's just to look at your phone for 10 minutes, but it's like 20. Yeah. That 20, 30 minute mark is about like how long you keep somebody who's attention per like completely invested without having some sort of a side tangent.
Ben: man. I think, I think I totally misinterpreted that method. Cause like I do the thing where I like to get over like procrastination I'll like set a timer for 10 minutes and be like, I'll just work on it for 10 minutes. And then I go and then I trick myself into going into like super hyper-focus mode and then the timer goes off and I like throw it against the wall and I keep working for another six.
Vyrus: like a, that's like a strategy they give like add kids to like do schoolwork as they tell you to like do it in 10 minutes. No comments. I think at the end of the day,
Ben: just outing me.
Vyrus: yourself, you got to do is not say that part. But I, I think at the end of the day, a lot of this stuff is tribal and like, nobody really has figured it out.
I mean, I remember right after the pandemic started. And like the first thing that happened at work is like all of a sudden I got forced joined into a slack room with like maybe a hundred. And it was totally silent for about 20 minutes until somebody came in the room and said, so you're all probably wondering why you're here.
You're all here. Because at some point on your HR profiles, you have indicated that you've all actually been remote before the rest of us. Nobody knows how to do that here. Figure it out, teach everyone else. This is your new priority.
Ben: that's very cool.
Vyrus: You're like I'm into their credit. It
Joe Fitz: let me guess that was a person who doesn't know how to do remote.
Max: Yeah.
Vyrus: I mean, yes, but it also,
Max: like a nightmare.
Vyrus: like you're not wrong, but also it turned out. They were totally right. No one else at the entire company knew how to do remote and like too much their chagrin. Like we laughed and we joked and then like, we wrote some docs, but like, it kind of worked like, it, it actually turned out like between our powers combined of people's random little, this one were your tricks that they'd all adopted for working remote for five, 10 years.
Like it, like, it kind of did like make the difference. You know? I mean, there were like, we do these things where like they have these polls that they take to kind of, you know, kind of take the temperature of the company every once in a while I got some companies that are kind of, you know, good about this.
Some companies are bad. Like ours is really aggressive about it. And like, I mean, you know, th the data doesn't lie, I suppose, like people really found
Ben: I mean,
Vyrus: quick, a little like to do is helpful.
Ben: You're acting like it's more complicated than just swapping the cream out of your coffee for Bailey's. You know, this is not, it's just that simple, but
Vyrus: I think it's a lot of like, explanation like that. Like there's a lot of like, there's like, it's the same advice that's out there if you Google for it. But if you Google for it, that advice comes with no explanation. So people don't really know how to internalize it and operationalize it.
Right. Like my favorite one that I love that I see, which is terrible advice, I think is the, like always get up and take a shower and put on like work clothes when you go to work. Because like that's the way to do it. And like, I, for a number of reasons, I think that's a horrible idea, but what they're the reason they're actually telling you that, right.
Is because like, and there's tons of neuroscience to back this up. Like you need some kind of mental divider between I'm awake versus I'm at work. It doesn't matter what that divider is. Really. The advice is just find what that is and do it. Right. And there's a bunch of little things like that. That's like, oh, like if you explain it, people will do this.
Ben: well, so maybe, maybe a you should make an online training about how to make online training. Joe. You think there's a market for that?
Joe Fitz: crossed my mind. I could set up, I could set up like a page and I give you like a teaser. And like you get the first court first lecture free. And then I start spamming you email to come down my sales funnel and learn all the secrets of, of training online training and how YouTube can be as successful, uh, you know, stay at home, work from home, teach people how to hack hardware trainer.
Ben: Well, you're not hardware specific,
Max: Don't don't
Make millions.
Ben: yeah, you gotta
Joe Fitz: millions.
Vyrus: going to say, this seems like the kind of model where like, instead of giving the training on it, you just have a consultancy. That's like, I will teach you particularly how to make online trainings for a lot of extra zero.
Joe Fitz: Yes.
Ben: so are you doing this through a platform or are you just cut your, I know you're like shipping people a box or are you hosting the lectures yourself or you're using some kind of a middleware.
Joe Fitz: So I looked at a lot of the middleware. options and they are all in those two categories, the like sales funnel and the, um, corporate compliance training. And, you know, there are others in between, um, that are not too bad. Um, and I probably could have found a platform that I would have liked and it would have been worth.
Whatever fees there are for those platforms. But, um, I had talked to a few people who are doing similar things and ended up doing, you know, WordPress will be, um, with LearnDash, which is a learning management system for WordPress. Um, and then WooCommerce, which is a store for WordPress then. And they all, all three of those work together pretty Well, Um, you know, I'm running the risk of running WordPress, but you know, I'm running the re I think I am better off running WordPress than I am. Some of these shoddy, like corporate, uh, oriented, you know, services that, you know, you pop, you pop that service, you're getting 15 corporate, you know, clients and all their email addresses.
Whereas you gotta get a PO my WordPress server to get my client list. Right. so you know, it's, you're always running a risk and, you know, I really appreciate it. The face-to-face classes where I really didn't need any need to know who was in my class. Right. I actually had stickers that I put on things. It didn't say like, my name is, it said, you can call me because like, it doesn't matter what your name is.
Right. Sometimes people want to be called by their handles. Sometimes they want their names. Sometimes they show up and they, they pick a new name, um, whatever. Um, and so I kind of long for that level of anonymity that we can like have, uh, but at the same time, uh, yeah, so I've, I've got that set up. It works pretty well.
I post the videos and the stuff on there. It looks pretty good. Um, there haven't been too many complaints. I thought about, yeah, I had thought about doing, like putting it all on a USB drive, but I was a little bit wary of that since this is, you know, my, my business and putting my whole business on a USB drive that I ship around the world was a little worrisome to me
Ben: Well, so just a to plug the site. So the training course is available on a
Joe Fitz: learned about securing hardware.com.
Ben: learned about securing hardware.com.
Joe Fitz: Yup. And you can just click and put in credit card numbers and get hardware showing up at your door and learn how to use it. Okay.
Ben: Uh, that's pretty cool. So, uh, what else is in the box?
Joe Fitz: So there's two classes. Now, the box has tools and targets. Um, the tools are Tigered. We talked about before, but magic. We talked about before a multi meter and a USB microscope and a few clips and cables and stuff. Um, the targets for the first class, the one target is a wifi router, um, which has been opened up and with some modifications made to make it all easier.
There's no soldering needed for the class. Um, Joe grand cover soldering in his class. I don't need to overlap with that. Also. I feel like there's some people who are scared of soldering, which is entirely fair. I avoid it when I can, I, I know how to do it. Um, so the fact that you can go through all these classes and do all sorts of hardware stuff without touching solder, um, I think is something people, people don't even realize.
Um, and it doesn't, you know, beyond this class, you can do that in real devices. Um, for the advanced class that just is online. Now that one focuses on like a hardware pen testing technique. So there's an additional board you have in there, a little, a microcontroller board, but then two targets. There's an SSD.
You got to go and, uh, open up examine, uh, black box analysis, you know, make a block diagram, do some threat modeling and then dump the firmware off of, um, and get jaytag working. And then there's a thermostat that you do the same thing. You start from beginning, you, you know, uh, look at the marketing material that comes with it, uh, do with some threat modeling, figure out how it works and how it doesn't work and how to bypass the protections they put in there.
Ben: Very cool.
Max: wondering if you have any like, sort of tips or like tools, uh, for people who are like trying to build their hardware lab out, uh, you know, if there's anything that you'd recommend the sort of like this thing sort of changed how, uh, how you, how you do a
Ben: Honestly, the USB microscope is interesting to me. I've still been using just those handheld magnifiers with the lights and them.
Joe Fitz: So, um, the USB microscopes came out of a class that I was teaching where the light in the room was really bad. Um,
that venue just, it was like, Fancy. It was like a wood paneling in the room, but the lights were just really dim. So it was really hard for anybody to see what they're doing. So, okay. I need some sort of lights in the kit and I had these little like, uh, uh, lenticular, like panel magnifiers in there before, but they didn't really work very well.
So I got a USB microscope and it was tiny. It fit in the kits. Um, and it plugged in over USB and you open up the webcam software and you could see great. And it had a really good focus range. Um, and it turned into like the thing that people like, oh man, this is the best part of the whole class. Um, best tool in the whole class.
Uh, they, they left, I liked the hardware stuff too, but, you know, um,
Ben: This is going to be one of those podcasts where I like go and buy a bunch of stuff right after.
Joe Fitz: so, um, yeah, let me, I'll send you one because I don't use, I don't sh I ship out a different one that I was able to get in quantity, not the one that I used to use for class, but I have a box of them that I used to
Vyrus: Oh, look at that. It's a podcast where you're getting free stuff.
Joe Fitz: no, not all of you listeners, I'm talking to you, Ben. Uh, so,
Ben: deal.
Joe Fitz: but, um, yeah, so you can also, the other thing is you take a picture and then you can work on a still image and counting pins on a still image is so, so, so, so much.
Ben: Yes.
Joe Fitz: moving things in three-dimensional things. Um, so yeah, That's that's probably the one addition that I didn't really expect. And once I added it made all the difference, otherwise like the pair of a logic analyzer and an IO board really works for me. A lot of people ask about the bus pirate and the bus part is this old Swiss army knife that like, it can do anything that's true, but it doesn't do anything really well.
Max: No.
Joe Fitz: so I rather have a logic analyzer that does logic analyze it really well. And an Iowa board that does IO board really well. And even though I have two tools, um, I feel it's a lot more flexible and versatile, uh, for a lot more things than just having that, the bus pirate. Um, so the tiger and bit magic is what I settled on again, uh, before the tiger and I was using ADA fruit's breakout board,
Ben: I use those.
Max: Hmm.
Joe Fitz: they work great.
Um, but I wanted something that had level shifting, built it in, built in as well as the head has already sought it on. So I wouldn't have to say to them before I hand them out tickets
Ben: Yeah, the problem with the eight or free boards is they just have like anonymous pin headers. So you have to, um, you spend almost all of your time, uh, like worrying that your wiring's not right. Or bumping a wire off and being like, shit, which pin was that on? And like, they're just, you know, numbered. Um, it's a, it's a struggle.
Joe Fitz: made for the ADA fruitless. So that they were labeled. So, you know, it said Jade, it said Tedia TDI and GTX and everything else. So
Ben: Oh, that's
Joe Fitz: on the, on the heat shrink tubing on the end of the wires set, all that stuff.
Max: So, yeah, I saw that you have a have that on for the T yard.
Ben: Yeah. I'd probably just get a T guard at this point.
Vyrus: would you, would you rec would you recommend that people like, I mean, I I'm asking for myself, I guess, like, is it a good idea to just get in the hardware first? If you think this is going to be a road you're going to go down or is it better to just kind of buy the thing or attain the thing that is your project, and then let that deal at your tools?
Cause like with software it's more like I just get the thing and then I find out, oh, I needed this. I need that. But that's cause you can just download stuff. Right. But with hardware it's a lot like.
Ben: No matter what, uh, oh, for hardware, you're always going to need a multi meter. You're always going to need a logic analyzer and you're always going to need an on. At like, it's kind of a minimum. So like there's other esoteric stuff you might need to bring in. Um, but the IO board is like, it's a serial adapter.
It's a J tag dongle. It's an SPI adapter. It's an ITC dongle. Um, I'm just alternating dongle and adapter for no particular reason. Um, but th the, the logic analyzers. So you mentioned the, um, the bit, uh, it was a bit magic, the bit magic to your logic analyzer. Right?
Joe Fitz: So I used to use the Salian logic
Ben: I love the Saley. I'm a, I'm a, I'm a salient
Joe Fitz: have pretty software and what's great about the same way is I could put that in front of someone who's never used a logic analyzer and they could fix.
Ben: Yeah.
Joe Fitz: of a salient is that they just continued the logic for which was their low cost for one.
And I think the low-cost one is like, it's bad for me because people, I tell people, you don't get four, but look at the eight because it's it's, you live it more, but you can get more. And it was a good upsell, right. But they just confused the four and they increased the price on the eight and now it's $400. So their cheapest product is 400 bucks, which is just, I mean, that's more than everything else in the kit combined.
Ben: So I, I love, I love sailing and I have one of the original pores. Uh, I also have one of the newer ones cause they're, they're, uh, they work with much higher speeds. I mean, there are more, they are more expensive, but they're, they're much higher speed. Um, but, uh, you can get Saley clones from China for like $7.
Uh, that'll work with their software if you kind of flip some bits here and there.
Joe Fitz: so the, the bit magic has actually the same design as the salient clones, which has the same design as the Saley eight, which is the same design as the USB and like several other, like th this is a whole, there's a, a Cypress application note about designing a logic analyzer using their chip, right? So, uh, they're all clones of each other.
Um, but, uh, pulse view is a, uh, very highly German influenced, uh, user interface to SIG rock, which is a very highly German influenced, uh, logic analyzer software package, which is all open source, which is great. And, um, it was actually October, 2018, which is. It had hit the feature parody I needed for my classes.
And that's when I was like, okay, that's it. I'm switching. Um, and so when I had the opportunity to go refactor and I was rewriting all these labs, that's when I'm like, I need to go fully in, on the logic analyzer. So I used a bit magic, uh, because it's, it's inexpensive, it's a little, a much higher build quality than the $7 clones you'll find.
Um, but you know, you could use either one. Um, but Yeah. it works.
Ben: Yeah. So someone, just, someone just showed me Paul's few, uh, like a couple of months ago and it really is, it, it is very similar to the sales interface. Um, but, uh, I think it it's like the fact that it's open source and you don't actually have to like pirate software to use it with, uh, the inexpensive Cypress board is, is pretty nice.
Um,
Joe Fitz: and once you use, when you use pulse views, uh, protocol analyzers,
Ben: yes.
Joe Fitz: when you see those, the first time you be like, oh my gosh, like, why was I wasting my time with them was salient. Like, they're beautiful. They're color coded. They're visual. They're easy to read. Um, whereas salient, I think their newest software is a little better, but it was like tiny white text on a light blue background that you couldn't scale.
And it's just like,
Max: players because open source software is never known for having better UI.
Joe Fitz: So yeah,
Ben: I a long enough timeline it'll get better. Right.
Max: Yeah.
Joe Fitz: post views protocol analyzers are beautiful. Um, and they're, they're part of the reason why I w I, if my first choice for anything.
Max: I imagine there'll be much more expensive. I mean, that's one of the things I never understood why there's just so many competing. co-source, uh, even when like the companies themselves seem like they're sort of, know, supporting the open source community. It's like, if you use the open-source software and get that working with your, uh, and like extended to work with your, your software, you never have to support it again.
You have to have that reference for the software and the number of signals and protocols that you'll be able to decode is going to just skyrocket.
Ben: So I wonder if I can use my actual Saley hardware with the policy software. I should give that a shot.
Joe Fitz: use the original one, but not the newer ones.
Max: Yeah.
Ben: Fair enough.
Max: About a, uh, digital, uh, logic analyzer. And I was like, I just sort of expected that to work with the open-source software.
Uh, and it was.
Ben: we got, we got, uh, the, the basic kit, um, the, the, the logic analyzer, uh, you got bit magic. You, um, everybody needs a multimeter for at least identifying ground pins. Um, and then you've got, uh, the USB microscope, uh, which is a new tip for me. That's cool. Um, and the IO board, uh, the Tigard, uh, which is that, that used to be like, you used to need like $3,000 worth of separate dongles to do all that crap, but through the magic of the Ft, 2 30, 2, or 2, 2 32, whatever you're using, um, you can just get by with one chip now, which is super nice.
Um, is there anything outside of that, uh, uh, that you'd recommend for beginners, like any other tool tips?
Joe Fitz: So, yeah, the core is the multimeter, a logic analyzer and IO board, right? When you've got that, you can do 90% of what you're gonna need to do. Um, the Mo USB microscope is a good addition. Um, and the question, uh, earlier was like, well, do I get the hardware first? Do I get the tools first? Well, once you've got those three, then go and get a target and start messing with it and decide what it is you need.
Cause the
Vyrus: and on, and then that just real quick in that vein, I was my followup question. And this seems like it's just too good of a segue of it, but I totally encourage you to finish that answer. Is, are there targets that you would encourage people to avoid because. One of the things I run into with a lot of people asking me about binary, exploitation is they go like, oh, I'm going to have to world of Warcraft.
And I'm like, okay. So I'm not going to tell you no,
Joe Fitz: Yeah.
Vyrus: like,
Ben: There's a,
Vyrus: you to get disappointed 10 minutes in.
Ben: th there's a book on that you should read first.
Vyrus: Exactly.
Ben: Yeah.
Joe Fitz: next thing you you're going to end up needing is a protocol specific adapter. So you're going to find a device that whatever uses its own, you know, jaytag software and hardware, and it's just going to be worth it to buy the vendor's device for that. Because if you're going to do it a lot of work with it, just, just do it like it'll, it's worth 50 bucks to get a piece of hardware.
That's going to save you five hours.
Max: Yep.
Joe Fitz: Um, the other thing that I would add, um, Is, an, uh, a microcontroller board and that doesn't have to be anything fancy, like a little $5 Arduino pro micro is all it takes to go and bang a few protocols that like your IO board can't speak or to do, like jaytag searching a regulator style just a little bit rougher around the edges
Ben: Is, is a protocol like the, the, the, the, the debug, the in circuit, debug protocol for some of the really tiny chips, some of the tiny Arduinos, um, there isn't really a good dongle for, um, but you can bit bang it with an, with an Arduino really easily.
Joe Fitz: tiger. Does it?
Ben: I stand corrected.
Joe Fitz: Can I get, it's got a little switch on it. You switched the jaytag SWD switch over to SWD and you're in SWD mode.
Yeah,
Ben: okay. I gotta get me one of those.
Max: Yeah. And, and SWD is a, is a very painful protocol. If you just want to try and implement it on your own,
um,
Ben: is
Max: much like J tech itself, if you, uh, are not using the tools provided.
Ben: there, there is an Arduino sketch that does it though. Like you can just drop in.
Vyrus: Yeah.
Joe Fitz: Um, but yeah, my controller board is really handy to, to, to get those protocols that either you don't recognize don't know, or if you want to do something like, uh, one of the labs I have you do is you basically intercept a UART transmission, right? And the microcontroller reads a byte in checks it. If it's a certain packet, it goes and modifies it.
Otherwise it passes it along.
Ben: Ooh.
Joe Fitz: do that in a few lines of C code on an Arduino board that costs five bucks and plug it inside in line in a system powered by the system. And it's a standalone hardware implement. Um, and that's one of the labs in that later class.
Ben: Wow.
Vyrus: Nice. So yeah, like targets, is there a way to like figure out if a target is like, I shouldn't go after that cause that's going to be too hard or.
Ben: I, I usually, I mean, I usually tell people to go in the Goodwill and buy the, the, the bigger, the bigger DSL modems or the bigger cable modems, because that means there's the old ones, the big old ones.
Joe Fitz: They're they're bigger. They're easier to see. They're easier to clip onto. They're cheaper. You're going to break stuff. That's that's another thing is like buy multiple because you're going to break it. And if you're not breaking hardware, you're not trying hard
Ben: we used, we used to say one must die.
Joe Fitz: Yeah. So like, yeah.
Ben: doing a hardware security assessment, we'd ask, we'd ask, we'd be like, we need at least two, you know? And then if they were like, sure, no problem. We'd be like, can we get three?
Joe Fitz: when someone hands you the cheesy, like encrypted USB drive that has their, a Bitcoin wallet on it, right. You don't break that. But you get a bunch of similar ones and you break them and when you know how to break them, then you can learn how to not break them. And when you know how to not break them, but then you can touch the one that you can't break.
Right.
Ben: Yeah.
Joe Fitz: Uh, so it's like, it's like falling off a bike. Right. You learn how to do it. So expect to break stuff. Don't, don't worry about it. You're going to break it and then figure out why you broke it and don't do it again.
Max: And I think for a target like that you'd want to avoid initially might be a bit more specifically. We want to target a consumer grade hardware, I think, as opposed to maybe.
Ben: Well, industrial is typically easier than
Max: Yeah. That's a fair point.
Ben: because you were talking about things that are still running windows XP, like in dust.
Joe Fitz: game consoles,
Max: No phones, no games. Consuls.
Ben: Well, unless it's a PlayStation, right? PlayStations. No problem.
Vyrus: nothing where there's already a dedicated community of people that are reverse engineering it for profit, right? Like what's the car thing, like ECS is probably, probably not a good first target.
Ben: Well, I usually tell people to, uh, before you, like, before you hack anything, Google it, right. Just Google, whatever the thing is and the word hack and see how far everyone else has gotten. Right. Um, and cause it's usually, um, it's usually very, I mean, it can be interesting to like reproduce other people's work.
Um, but, uh, it is, I think it is a mistake to like launch in, on a new hacking project and decide that you're going to do everything yourself without looking at the internet. Um, because you're basically going to spin your wheels on, uh, you know, trying to identify some chip that like doesn't do anything useful, you know, for three days.
Um, and
Vyrus: that's, cause that's basically what I'm thinking of is like the roadblocks, right? Cause like, like one of my lingering projects has been like, there's a lot of, uh, uh, IOT based like security camera stuff. That's been coming out recently. And a lot of that stuff is like tied to the vendor through like the controller system and like without going into details because you're all smart people.
It doesn't need to be right. And it's like, okay. On the one hand, like that tech seems simple enough that the manufacturer probably wants to make it really cheap. And so it's probably not like a phone. Right. And I probably don't need to decaf chips to figure out how that thing works and smash it and just like swap out firmware and have like a running like open-source project of like buy all the good cameras, then nuke all the firmware and put your own on there.
And don't have to pay four times the price for an IP camera, or maybe they're all Isaacs and I'm wasting my time. Right. It's like, you don't know till you get one.
Ben: Yeah, that's true. Well, I gotta say my I've been noticing that a lot more, uh, embedded devices are Android based these days. Um, and that's actually really fun, uh, because basically you just need to get a serial connection to it or a USB connection to it, and then figure out how to enable debugging. And then you can use like the Android debugger and like hack it, Android style and the best.
So, and granted, there are phones that use Android and they're going to be harder targets and it might be more locked down some of them. Right. Um, but there's just random devices that have no threat model at all. And literally don't really care. Like for example, um, I found this projector that I found, uh,
Vyrus: are, there is a whole community that I'm part of on discord that propped up over the last month. That is nothing but
people
Ben: Yeah.
Vyrus: mutual mutual acquaintance of hours, hours buying projectors and the channel is called serial killer.
Ben: Yeah.
Vyrus: doing is serial attacks and dropping ADB and
Ben: Well there's
Vyrus: consoles,
Ben: there's no, there's no need to attack it, man. Like, like I got one of those projectors, cause I'm in, I am in that channel and it was like, oh, it looks fun. So I ordered one of those, uh, and it shows up it's a, it's a pretty nice projector and you pop the top off. And the, the serial pins are just labeled.
I was like, well, there's TX, there's RX sweet. We'll just connect the serial cable. You, uh, open a console, boom root shell. Right. And then you're like, oh, ADP's enabled. Right. You can just like, you know, connect to it with a visor and a, and see what it thinks its UI is supposed to look like. Right. And then they have no threat model.
Like they don't really care if people hack it. In fact, if they knew we were, you know, if they knew we were doing this, they would probably add that to their, uh, you know, their, their, their advertising. Like it's totally hackable, like, like have fun. Right. They don't care. There's no, there's no vendor lock-in software on a projector.
Like it's just, um, you know, like they have no reason to keep people off it, but this is the second device I've seen in a month. Um, that was Android based and, and, and, uh, in a kind of surprising way. Um,
Joe Fitz: uh, and an indicator of like a device that is like worth looking at from a hardware perspective is one that is cost optimized, but not too cost optimized. Right? so
you want something cost optimized, meaning it uses off the shelf components. It just copies the manufacturer's reference design and throws a couple options of components on there.
And maybe has shoddy software, maybe not, maybe he's has been like OEM software versus to cost optimize, which is a thing where they actually designed it for mass market and integrated everything and actually custom made everything. Right.
Ben: I have an example. So I,
Joe Fitz: not too cheap.
Ben: yeah, I, I have an example of a two cost Austin optimize thing. You go into a Walgreens and you see those little handheld console things for $10 that are like a handheld Nintendo. And they come with like 30 or 40 games in them.
Max: The black
blob of a
Ben: Exactly.
Joe Fitz: a chip
Ben: Yeah. So there's someone made an ASIC, that's a S on a chip and they make these little consoles, but, and I got really excited cause I was like, I can open it up.
I can like get in the flash. I can put my own ROMs on there. You know, I can play, uh, you know, Mario brothers or like some actual rum, uh, like on a thing I got for $10. I was super excited. I opened it up. Um, there's nothing on the board, but a single blob of black approxi right. And then the wires out to the buttons in the battery.
Right. And that is, that is, that is, that is actually a fabrication technique that is so cost optimized. The, yes, the thing is an ASIC. They put the whole thing in one chip. Right. But they didn't even bother to purchase packaging material for the chip. They just produce the raw chip, soldered it down to a board and then covered the whole thing with, uh, with that black goo, um, that you, you can't, uh, you can't take off without destroying the chip underneath,
Vyrus: Yeah.
Ben: you know?
Max: the finance was like, what? A 65 0 2 or something like that. So somebody has a project to literally recreate a 65 0 2 on a breadboard.
Vyrus: I've seen so much blood goo.
Ben: Yeah. I hate that. Black goo, Hey, you got any tips for removing the black.
Joe Fitz: Um,
Ben: It's okay if you don't. Yeah, just throw that out there. Hey, okay. So we were talking about,
Max: got solvents for that.
Ben: oh yeah, really? Do they, will they, will they melt the bones out of my hand? Uh, yeah. Let's not talk about that.
Vyrus: Yeah. It's one of those chemicals,
Max: It's it's it's it's it's it's not HF. It's it's brake cleaner and a couple other things.
Ben: Uh, if, if people are interested, you can look up the things I won't work with blog or whatever
Max: It's
on list. It's
Ben: want, I don't want to get into, I don't want to get into these types of lawsuits. I don't want, I don't want people sending me pictures of like boneless fingers, like why
Max: there
is some flooring, but
Vyrus: a C I was going to
ask,
Max: it's it's not an acid. It's
Ben: only a little fluorine.
Vyrus: I was going to ask, is it more or less dangerous than hydrazine?
Ben: Uh, well, that's, that's a low bar. Hey, so, okay. We were talking about like how to find an easy target, um, which sounds real creepy when I phrase it like that, but, um, uh, I am interested. So all, all, like the reason we're telling people not to go for newer boards is because on newer boards, everything is tiny and unlabeled.
Right. And so, um, I, uh, I have kind of shaky hands to start with. Um, and we kind of were mentioning the, um, you know, the PC bite probes and stuff like that earlier, but I'm wondering if you have any tips for connecting. Probes to kind of the newer boards where like, I've tried to solder mirror wire to these really small pads.
Um, but if you screw up at all, you basically lift a trace. You know what
Max: magnet where?
Ben: what'd I say? Mirror wire. Yeah. No that's for hanging mirrors. That's much thicker. No, no, yes. It wire my bed. Yeah. Magnet wire. No, a soldering mirror wire to things. That'd be insane. Yes.
Joe Fitz: um, I try not to solder, like, so, so Yeah. the, the like, oh, there's a thing I'm going to put a probe on it. Right. That's great for doing it one time. Right. But I'm going to need to connect to this spot a few times. So I want to solder a wire onto it. Right. And the next he says like, oh, well, if I saw it on the wire onto, and I accidentally tug it, I'm going to like peel it out.
Right. So I'll, I'll put some hot glue over it. Like that's the next step people do.
Ben: Oh yeah. I've done
Joe Fitz: know, that's fine. But like, then there's hot glue everywhere and it's just annoying. Um, so one approach is, um, I get a strip of 0.1 inch headers, and I glue that. to the edge of the board. Right. And then I use magnet wire from the test point to that 0.1 inch header.
And I just like populate all of those pins of that header and you get it.
Ben: Oh yeah. That's tomorrow.
Joe Fitz: you know, like, oh, okay, I'm going to probably need about 10, 10, 10 test points on this. So I get 10, a one inch header, 0.1, your tenders in a row, glue them down. And then just solder wires to the ones that I need them. Then I have a header that I can stick onto.
I don't have to put any tension on the solder pad itself.
Ben: is, so that is so smart.
Joe Fitz: If I'm going to use it across different devices though, I go straight to a jig, right. Where I measure its location and I drill a PCB and use a poco pin.
Ben: Ah,
Joe Fitz: So.
Ben: man, I wish he told me about that header thing like years ago.
Max: Yeah. I was like, I
used that when I first met you.
Ben: Yeah. You have no idea how many times I've looked Stoddard, a mirror wire Suzanne, but I did it again, a magnet wire to someone tiny little pad and then, uh, you know, glued it down and then like tripped over a wire or something like a while later and just like ripped all the traces off the board.
Max: Yeah.
Joe Fitz: there's a class that I don't teach very often, which is the physical tax on x86 class. And, um, one of the targets is a, a windows tablet, and that is the cheapest windows tablet at the time that you could get, you could get them from Microcenter, it's the one that you go to micro center, like, oh, there's a windows tablet for under a hundred dollars and I'll buy it and you buy it and then you realize it sucks.
So you go and you turn it. And you're like, oh, for only $50 more, you can get one that doesn't suck. And so, you know, they always had tons of, um, of, of these like an open box. So you just go in and you buy 30 of them. Um, which is how I got from my class. Um, but what we had to do is, you know, I have people poke around on this to find the spy pins, to find the, uh, the firmware to dump the bios off there by hardware.
Um, I squared C is how the touch screen controller was connected, um, and a few other interfaces in there. But the problem is, you know, when you poke around inside a device that has a battery. It's really hard. Like you can't turn power off. Right. The battery is literally soldered in. Right. so I was worried, like everything's just going to break because people are going to like, be shaking with our hands and poking the wrong spot.
So I did is all of them. We repaired with, with magnet wire, we solder to the points and we put it all to a row of headers on the edge. And then we cut a notch out of the back panel. So we could snap these things back together. And you had a tablet that was closed with a row of pins coming out the bottom side.
And that row of pins was you are a spy. And I squared C connected inside the board. Right. And it worked great. Like I use those habits over and over again, and very few of them died.
Ben: well, Uh, I think the version of the, the class that I actually went to, you will use those tablets. I vaguely remember something about this. That is a, that is a really good trick. So I wasn't, I wasn't totally far off with the magnet wire anyway.
Joe Fitz: Yeah. I mean the, the peeling of traces with magnet wire though, everyone's
Ben: Yeah. It hurts. It's a certain kind of trauma
Max: The, for like two, two days I work, I was trying to get, uh, these DuPont headers, uh, 'cause like a DuPont headers is like the most annoying thing in the world. So I was just going to, you know, cut, cut the cables in half inside of them, to the points I needed. Um, and cause it was actually broken out like so it and the, the, the, the the wires themselves, which is not take any solder at all.
I, I think they were aluminum wires or something, but I've never experienced this, this, uh, the, the most frustrating one at the world. And I literally wasted like a day and a half, uh, just, just trying to to this over and over and like
Joe Fitz: So a solder pot,
Max: up all the solder, like
Ben: had that I've had that
Joe Fitz: tending wires is well worth it. So instead of like, instead of a soldering iron and I strip a solder you get a little pot.
like crucible and you put like half pound a solder in it, and you can put all your solder scraps in there too. Um, and it just melts since you have this, this bubbling pot of metal and you just dip the end of the wire in and I tins the wire, so it has solder on it.
And it's so much easier to go and solder it in place then.
Max: It was a good for a attending the, uh, Southern and to
Joe Fitz: I've never had much of a problem with the attending. I have a tip Tinder that I use as well, but yeah, same,
Max: Yeah.
Joe Fitz: deal.
Max: nice.
Joe Fitz: And also a trick I've seen. I've never, I haven't tried. It is, you know, you get those, those ribbon cables and let's say you have to solder like a whole bunch of pins and you have a ribbon cable.
That's the right pitch, but you want to strip it. Well, it's really hard to strip, you know, 20 wires altogether. Um, put them on a laser cutter though. And the laser cutter will cut right through the sheath, but not the metal. And then you just slide that end off and then you have all your wires lined up, just the right.
Max: Oh, that's pretty amazing.
Vyrus: I've seen like some super crazy pictures of people using like, uh, like modified G circuit jigs. Like basically they opened the back of like a DVD burner and put like, and like you put funky firmer on it. And it turns the laser from the burner into kind of like a and they just run over chips and it totally just takes the enamel off perfectly
Joe Fitz: Well, so I did that for obscuring chips. Um, I've got a, like a five watt laser on top of a CNC machine. Um, so you know, to hide, hide the chips. So you have, it makes it harder for class. So, you know, you have,
Max: Nice.
Joe Fitz: it just stinks a lot.
Vyrus: I just thought it was cool, but like, he didn't need some crazy expensive laser. It was like DVD burner and some pirate firmware. Yo.
Max: Yeah, that's pretty, pretty awesome. I'll let you can, you can get, uh, the etching, uh, seeing like, uh, CNC rings for like 20 bucks on online these days to do
Ben: Yeah, but you have to, I actually got one of those for etching boards. Just the, to give it a shot. I got one of the, the Chinese ones. Uh, it was very inexpensive, but it shows up with absolutely no safety features. Like it, it doesn't have, I mean, also the reflow oven did too, but the, the what, the, the CNC etcher, it was a bit scary.
Cause it's basically a drill bed on an X, Y table. And it didn't, it didn't come with limit switches.
Max: Yeah,
the one doesn't even have a, doesn't even have a motor on it. It's just, it's just for UV etching.
Ben: Oh, I see.
Max: It's pretty slick though. It's just like a, to prepare sticks, but yeah, out of those, those things will have safety features.
Ben: So have you gotten into a decapping at all or is that kind of not your thing?
Joe Fitz: no, um, eh, not, not for the lack of desire, but for lack of time. So,
Ben: it does the same time consuming.
Vyrus: You having always seems like that thing, that like, if you don't actually have an E this is not a thing. Cause it's like, I've never like the one thing I've always found it difficult to learn. You know, I can, I can read enough anarchist, cookbook style, bullshit to like, you know, poor noxious chemicals. I shouldn't have on top of wafers and do stuff all day long, but like all the, all that.
So I can get to ultimately a grid that I can't read. Right. It's like, there's no key of like this line. Plus this line is like at this kind of circuit, like you have to just know what that stuff is at some point.
Max: Well, Joe, you were saying that was your history though. That was what you did over at, uh, until.
Joe Fitz: yeah, so, uh, you know, I, I was actually in a tools group, so we were writing software to interface to the debug features, built into the chips that would run on the testers that would connect to the pins and let them go and do the debug.
Max: Gotcha. awesome.
Joe Fitz: that lab, like we had the inner freedom missions machines. We had the, the laser laser assisted device device altercate alterations.
So like you shoot a laser and see if it fails. And that's where, you know, your bug is. Um, Yeah. So all sorts of neat stuff that like, he was really, people say like, oh, chips are getting smaller. Someday. All these decapping attacks are going to stop working. That's false because the chips would not work in the first place.
If we didn't have all these expect inspection tools, right? Like you'd never have a saleable yield of Silicon, unless you have the tools to do the failure analysis of the ones you manufacturer. And those are the exact same tools you're gonna use for the reverse engineering as well.
Ben: Yeah. I mean, they are sort of prohibitively expensive, but
Joe Fitz: that's a few have them full time or you have the current model or, you know, all that
Max: Yeah.
Ben: there are like third-party, uh, failure analysis labs in various places around. Uh, so you always have the option of renting time.
Joe Fitz: and especially in close proximity to Silicon manufacturers.
Ben: Yeah.
Max: It turns out those machines are really expensive to ship,
Ben: Yeah.
Max: they usually don't make it far off.
Ben: And actually there's, you know, there's so few of them that the people that go around and do the regular calibration, like they are familiar with the individual machines and they'll like, they'll give them nicknames and they'll like, talk shit about specific machines behind their back.
Sometimes if they're like trouble,
Joe Fitz: Yeah, those are the people who walk around with test equipment, running windows, XP embedded, and they go to AMD and they plug it in and they calibrate the AMD machine and they hook up to the network and then they go to Intel and they connect to the Intel machine and they're on the network. And then they walk to like every other Silicon manufacturer with their test equipment, running windows XP on the network.
Ben: Joe is, Joe has adopted the maximum supervillain pose right now. He's literally, he's literally stroking his beard.
Max: Yeah.
Ben: Yeah.
Max: So the fricking, uh, what's that called? the vibration dampening, uh, support each one. There's like four of them individual, uh, device, uh, connected over ethernet makes no sense. Makes no sense.
Ben: Yeah. So we were talking about like, uh, how industrial equipment is maybe not the hardest target. Yeah. Yeah. That's true. Hey, you were, uh, speaking of which, like you were messing around with SCADA at one point.
Joe Fitz: Oh, that was a while ago. Yeah, but I've already, I've already, you're a PLCs.
Ben: Yeah, that's right.
I'm a,
Joe Fitz: I think so.
Ben: oh yeah, no, I got, I got them back. I've just, I I've been afraid to, you know, plug it back into my new plant. Cause I, I don't know what kind of weird implants.
Joe Fitz: Yeah, actually, um, that was, uh, Interesting because that's when I actually made my first hardware implant before hardware implants were a thing. And then like, suddenly this and catalog came out like, oh, so people are really doing this. Cool, cool. Let me do more. Um, but yeah, like you open the thing up and it's, it's, it's over the shelf parts, right?
And there's no ups obfuscation of anything. There's no tamper evident anything. Right. You just literally open the thing up, pop it open. I was able to stuff an Arduino and our wireless module inside soldered onto the board closed the thing back up, even with the case off, you couldn't see, it was there, there were so many boards already in there, but yeah, like, so it was a, uh, Siemens, uh, seven, 1200, I think it was, um, uh, PLC.
Ben: Yup. Which is running at a bunch of like power plants, chemical blends, water control stuff. That's a,
Max: Oh, yeah,
Ben: that's a very widespread,
uh,
Max: common one.
Ben: yeah.
Joe Fitz: Well, and it's interesting because a lot of them are fine. Like, you know, if you are in one of those facilities, that's great. But if you have a remote station, that's monitoring water levels downstream from your hydroelectric plant and you don't consider the fact that this thing is readily tampered. And you've got a problem, right.
Or if, you know, you have this plant that's running for 20 years and then, you know, of course you want to check Facebook on the, on the PLC controller, then you put it on the internet. Like, you know, it's the same story over and over again. Everything's fine till you put it on the internet.
Ben: Yeah, they, um, they, uh, they talk a lot about air gaps and the power industry though. They're like, they're familiar with the concept of air gap. I heard a lot of people say like air gap is life. Like it was, uh, like a slogan they'd shout it. You know, air gap is life.
Max: And then somehow those, would still be on the internet. Uh, be like where air gap.
Joe Fitz: Well, no it's
Vyrus: I feel like
Joe Fitz: gap. Meaning, meaning it doesn't use wireless. We just put a wire.
Ben: Yeah.
Vyrus: the story of all air gaps is how no one in the entire universe apparently knows what an air campus.
Ben: Well, I, I knew that the, the, the term was starting to get overloaded when working at a cloud company and they started talking about their like cloud, their air gapped cloud. And I was like, I was like, I think there's been a fundamental, I think there's been a fundamental misunderstanding. Like they're like, yeah, it's like, there's two clouds and there's an air gap between them.
And it was like, okay, no, I.
Joe Fitz: Um,
Max: Laughing or crying
now.
Ben: I that
Joe Fitz: hear a lot about are our data diodes,
Ben: oh, well, that's,
Mitchell: say that.
Joe Fitz: And this
Ben: you know, that that's actually, that's actually a real thing. Um,
Joe Fitz: that's what makes me laugh.
Ben: so what w what they do, basically, they have like a logging. I'm not sure if this is what you mean by data diet, but this is actually a common trick.
And some of these networks where they're trying to make an air gap, right. So they have a, um, they have a logging requirement, um, from like the secure, you know, control network out to the rest of, you know, the, the, the corporate network or whatever, like to where the logging server is. And what they would do is they would take, um, like an ethernet cable and they would cut one of the pairs. Right. They would cut the, uh, the, the receive pair entirely so that you could only send, uh, data in one direction on this cable. And then they would switch to using a SIS log over UDP. Uh, and they would just like cram all their logs through a one-way ethernet cable, um, over UDP. So they didn't have to do a bi-directional handshake to start it off.
And I actually always thought that was kind of a neat trick. Um, but you know, it is, it is a little messy.
Max: Yeah. How many of those are actually cables that only have two pairs
attached?
Ben: Well, yeah, I mean, so
Joe Fitz: of them are plugged into more advanced network of doctors that have auto direction and auto pair
Ben: Yeah.
Joe Fitz: in, but they didn't at the time. And so they tested it and then they upgrade the network adapter. And suddenly it's not a data diet anymore, but it still works. So they don't notice because it works in one way, but they didn't test that.
It doesn't work in the other way.
Ben: I imagine that's happened
Joe Fitz: The other thing. So, you know, they have like the air gapped, so no USB drives were allowed in that's fair. Right. But then all this software, it requires dongles like activation dongles and licensed dongles. And they don't want to like have these walking around. So what they do is they get like a dongle server, right? it's a, basically a USB sharing server that they connect to the network to their aircraft machines. Right. And this thing is like a crappy, like embedded system with eight USB, one USB port in a hub, it's a raspberry PI, right. With a hub, right. Running Linux two dot four. And like some shoddy, like a USB over IP protocol with some constant drivers on the
Max: That's a spec. I believe.
Joe Fitz: end.
Vyrus: It specs net as a service.
Joe Fitz: yeah.
Ben: USB over IP, over ISD.
Joe Fitz: so you show up with your malicious bot, you show up with your malicious dongle, you plug it in this thing, you pop the USB server and then you have a direct connection to all these dry device drivers running on all the systems on the network, federal air gapped,
Vyrus: I mean, what's great about that is like, you don't need to even put your persistence on the system. You can just put your persistence on the USB dropper thing and have that it's like,
Ben: Well, they have it show up as a hid device.
Vyrus: yeah.
Ben: Oh man. Um, good stuff. So, um, a bit of a topic change. Uh, so there's one other thing I wanted to run by you. Uh, we may have to edit this out, but I was wondering if you've been following the, um, the work of the guy who's been, um, uh, dumping the Intel microcode out.
Joe Fitz: Have, um, and you know, not, not very closely, but I keep an eye on it. Um, it's pretty interesting to see all the stuff that's, that's being pulled out. Uh, none of it is really surprising to me though.
Ben: Yeah.
Joe Fitz: you know, it, uh, on, on the one hand, like, uh, I'm excited to see all the work that's happening and like all the proof that like some things that I thought were issues and were dismissed as non-issues a decade ago, are turning out to be issues.
Um, on the other hand, it's, it's a little frustrating cause like he talks so much about his progress that like there's always something new, but like, oh, w w what was the, what was what's what's new this time? So, um, but Yeah.
persistence. Oh my gosh. Persistence. I have no ability to ever, uh, uh, come close to.
Uh, so it's pretty impressive in that regard. Um, Yeah. And it's, it's all baby steps, right? You know, you, you, you get your foothold into the manageability engine, you get your foothold into the micro. You know, you take enough time, you reverse microcode. Um, so nothing, nothing that's going on is impossible. Um, it's all difficult and it all takes a lot of time and knowledge, but like, uh, I just wonder, he's focusing mostly on like one of the arm when he Adam cores, um, which as opposed to the, the big core, like the ones in the desktop server CPU's so I am curious whether there's more work that's not being published or whether, you know, other people are following along on their hardware at home or at their place of work.
Max: Absolutely
Joe Fitz: Uh,
Max: there is between the, the Adam. I think it's the has Wells and et cetera, and
like the well line onwards.
Ben: one other dude who wa who wrote, um, some kind of more general purpose assembly that was using, um, It was using a side channel through like the instructor. It was like the, the instruction counter or something that the number of operations there's like a, there was some register that was counted the number of, uh, operations that had happened. Um, and he basically constantly queries that to see how many operations had happened and then like runs, um, uh, different, uh, instructions and makes inferences about differences in the number of actual, uh, like microcode instructions that have executed when he like ran a certain bit of assembly. Um, so he's basically using the, there's a microcode instruction counter, and he's using it as a side channel to basically, um, you know, kind of, uh, specter out, uh, uh, uh, what the actual microcode is.
Um, and I th I thought that was pretty interesting. Um, but yeah, I don't know if that's been sort of made general purpose yet, but, uh, yeah, it's definitely, uh, I, I actually, that was one of, that's another one where I saw the guy's first blog posts, and he's like, I'm targeting this specific hardware and you can follow along at home.
So I went out and got that exact chip. I have actually, I have exactly the same debug cable he has in the same, uh, unit. Uh, but then I put it on a shelf and I haven't been back to it. So, um, uh, in the event I actually run, run out of other stuff to work on. I might have a swing at it. Um, just sort of re re like reproducing the steps.
Cause like, sometimes it's valuable just to like absolutely mimic what someone else did, you know? Cause
Max: When it gets to a certain point too, you could also potentially a fair in the CPU into a risk five machine.
Ben: awesome.
Joe Fitz: Sorry.
Max: Yes, certainly risky business.
Joe Fitz: I hear.
Ben: It was, it was absolutely required for that one mission impossible movie, right. Processor it's re it's really impressive. And now, and now everything's switching to arm, I guess the future is nothing but risk dead air, um, dead air. Okay.
Vyrus: No, because it's not funny only because it's true.
Ben: Everyone's like, yes, the future is nothing but risk. I'm like, uh,
Vyrus: Yeah. Basically.
Ben: well, I, I feel like, uh, the, the general blood sugar is dropping. So does anyone have any, um, let's, let's move towards a wrapping up, I guess. Does anyone have any, uh, other questions? They didn't get a chance to ask. Joe, do you have anything else? Uh, you'd like to plug you haven't you haven't plugged yet.
Joe Fitz: Yeah. Besides Portland,
Ben: Oh yeah. We didn't talk about that. We didn't talk about the hackerspace stuff. So,
Joe Fitz: you're in Portland area, there's a hackerspace. so once things are back to normal, like compass the backspace.
Ben: so let's, let's, let's get into the hackerspace stuff a little bit before we go. Um, so what's, what's the name of your hackerspace in Portland there?
Joe Fitz: It's a control H or PDX hackerspace. Cause my both names. Um, it's in north Portland. It's uh, Kind of about commercial building that was a bit run down and, uh, we kind of rehabilitated, or John and Melinda basically did all the work to rehabilitate it. Um, and now we've got Nice. big area that, uh, you know, in the normal times Is often used as a co-working space.
There's a craft lab, laser cutter, you know, what whole wood shop, um, an electronics lab with a PCB mill soldering stations the whole bit, I I'm the one PMP machine, but maybe, maybe this lined up there once it, has outlived its usefulness that, or takes up too much space in my office.
Ben: it is, it is the space, mostly hardware focused.
Joe Fitz: um, I guess so, um, it, it's, it's definitely a hackerspace and a Makerspace. There's a lot of members who are all about making things and really not at all involved in InfoSec at all. I know there are some hackerspaces that are like totally into the, the InfoSec side of things that the, that that end up hacking.
Um, but yeah, a lot of craft, a lot of woodworking, um, all going on there.
Max: Oh, that's awesome.
Ben: is that, um, I know, uh, that, uh, OSH park, the company I have, all my boards made out, got spun off of, uh, some Portland hackerspace. Was this, was this ad or was that a different
Joe Fitz: that that's dork bot dork bot is a co like informal group of, I believe it's people doing weird things with electricity.
Ben: Yeah. I used to go to dark, but Seattle, so that, that came out of dork by Portland.
Joe Fitz: my Portland, which recently has had their meetings at control H so
Ben: Oh,
Joe Fitz: their, their gatherings they're about overlap crews.
Ben: very cool. Thanks for clearing that up. And then, so how long have you been involved in besides Portland?
Joe Fitz: Um, besides Portland was the first info site conference I've ever at, and it was the first one I ever spoke at, um, its first couple of years. Um, but then about five or six years ago, it, uh, the, the old, the old crew was kind of getting worn out and, uh, we had a date and we had a CFP open, but we didn't have a venue, um, or any other plants.
And I, you know, training was a little slow at those in those days. So I had time, so I found a venue and like, oh, great job. Keep going. You're like, what? And uh, like, well you got the venue, you know, you can do the other stuff too. I'm like, uh, okay. And, uh, I ended up, uh, with it on my lap and you know, that was fine.
And it evolve of, uh, and you know, that was, that was when, besides Portland was a hundred to 150 people. Um, at the last in-person when we had, I think we hit a thousand registrations, um, turned into a very different kind of event over the course of five years. Um, but you know, multi-track multi day, very local focused, uh, free admission conference.
Um,
Max: I didn't realize it was admission.
Joe Fitz: Well, it's, it's pay, pay what you want. So you know, you can register for free and we'll there's room for you. Um, but the more money you give, the more stuff comes in your swag bag.
Ben: you either pay in money or guilt
Joe Fitz: Yeah.
Vyrus: The money or shame, right.
Ben: or shame, but every, everybody pays.
Joe Fitz: Sometimes I pay extra not to get the, uh, the corporate swag bag. Right.
Ben: That's a, that's a good option. Um,
Vyrus: swag bag, like instead
Joe Fitz: I just, I just remember like one of my early black hats, or maybe one of the black hats, I finally was like, uh, common off that I could actually look around and, uh, Norse Corp was giving out like Viking hats and stuff, and I'm just sitting like,
Vyrus: Oh, I remember
Joe Fitz: black hat. This is supposed to be a big deal.
Like this is, this is like the pinnacle of the industry. And I'm sitting here watching people come out in droves, dressed up like Vikings, and I was kind of embarrassed for InfoSec,
Vyrus: Yeah.
Ben: I, mean, who doesn't want a Viking hat though?
Vyrus: I had a similar experience. My first block. I know that feel.
Ben: Yeah. Black Blackhat was really interesting this year. So I, I actually went to Def con this year because it was like two days before it was, it, it was the last couple of days of the period where I w like I thought the pandemic was over. Right. Um, so I, uh, uh, and I'd given it, I, I gave a talk, which they, they, they made everybody prerecord, you know, cause they're doing like the hybrid conference.
Um, and, uh, so I guess I didn't really have to go. It was optional, but I, I, I kind of liked the idea of going to a smaller Def con one more time. And, you know, this might be the last smaller Def con that ever happens. Um, and, uh, I went down there and for the first time ever, um, you got a free ticket to black hat with your Def con badge. Yeah. Um, yeah. So the, the whole like black hat, uh, used to be like the, no, not just the business hall, all the talks.
Joe Fitz: really?
Ben: Yes. So like we know, I noticed, I mean, the first, the first time I noticed there's a couple of years ago when, uh, Def con, uh, was in Caesars where Blackhat used to be. Right. And although I was at Def con you know, everyone went to the, like the gallery bar, whatever, to hang out and, and, and I kept having flashbacks to black hat and I got so confused.
I would forget if I was at Devcon or black hat. Right.
Vyrus: The fact that they did that after Blackhat announced that they were going to drop the mask requirement for large portions of their conference was
Ben: They were just trying to get people in there because no one went.
Vyrus: I know. And like, that was their way to do it. They were like, it's cool. We'll give out free pack. We'll give out free passes and we'll make people not have to wear masks and
Ben: Like, so it's actually, I actually signed up for a free pass and then I didn't even go over there. I was like, whatever, this will get me into the, the, the videos or whatever later. But so we sort of making jokes, you know, when the first year, first year Def con was at Caesar's, we started making jokes like Def con is, is the new black hat.
Right. Um, but this year it finally came full circle. Like, like you get into black hat free with a Def con badge rather than the other way around.
Joe Fitz: I did, I did remote training for black hat, but there was no way you're getting me to go there. So.
Ben: Uh,
Max: I mean, that's, that's pretty impressive though.
Ben: I don't know. I mean, I had a pretty good time, uh, Uh, at Def con it's like they required everyone to have, um, uh, a vaccine card and then they gave you like a wristband and then they, um, you know, they wouldn't let, uh, anyone without a wristband into various parts of the, you know, the hotel, including the entire pool area for the whole weekend of the event.
Um, so it was, I mean, it was, it felt a little weird, like flying in and flying out. But when I was actually there, um, it was, it felt all right. Um, but the thing I really liked about, uh, Def con this year is that basically all of the people that were there were very, um, they were very dedicated, you know, they were, they basically did some like mental calculation, like, do I still want to go to Def con if there's some risk of death and decided, sure.
Why the fuck not,
Vyrus: Yeah, I probably, I probably would've gone if it weren't for the whole like, oh, I have an infant.
Ben: Yeah. Yeah. Well, so it will, you have a reason not to die right there. So, but the other, I mean, the other wrinkle, uh, was none of the party crews showed up, like, uh, one of my friends who's a gun and said like, yeah, none of the lucky loser here this year, but also none of the party people. Right? None of the people that organize all the big parties.
Um, so there were like almost no parties. Uh, and as a result, all of the games were incredibly well attended. Right? Cause people were like going to bed relatively early and not waking up, hung over. Uh, so like the tele freak challenge and like all the challenges, like people really hit hard. Um, so it was kind of like a big year for that.
And the badges were really cool.
Vyrus: Yeah, that was a interesting, like how that rolled out. Cause it was like completely new people. They had like what, three months to throw the whole thing together. And Jeff was like, here's a bag of money, like everything's shorted, but whatever, figure it out. And like, if you don't suck, maybe we'll call you again next year.
And they totally just knocked it out of the
Ben: They totally killed it.
Vyrus: and, and was what I thought was hilarious is like, what's, you know, for the past, like what eight years, what's the one experience that like everybody at every level has dealing with Def con is it's OAI went to Def con and got a paper badge, right. Because like heaven forbid like when there's not shortages, like deaf God know how to, you know, stock badges.
Right. Because that's just like, that's the problem everybody expects now. And what's hilarious is that like, and, and in the land of mass shortages, worldwide of everything electronics most especially right. Pretty much all the speakers, all the goons, all the like CFP review people, all of us like back hand stage people that like, didn't go all that badges in the mail.
So I'm like, oh, this is the secret to badges working. Even under the worst of circumstances, we just don't let Jeff handle them.
Ben: Yeah. Uh, yeah, there were plenty of badges, like, so, so I, uh, I did a workshop and gave a talk and I opted for badge payment for both of those. So I ended up with like six badges or something.
Vyrus: Yeah.
Ben: Um, and, uh, uh, but they're, they're cool. Cause they have like, they each have clicky keys on them.
Vyrus: And they're like cubes, you like hook them all
Ben: Yeah. They're cubes and you can chain them together and they can add keyboard.
Right. So I can actually chain all six of them together and use it. Like you can play a multiplayer Simon game, uh, with them, but you can also use them as like a, uh, I can like plug them into my DJ setup or something and just have a truly bizarre, uh, uh, mini controller.
Vyrus: Streamer rig. Right? Like,
Ben: Yeah. There you go. Ah, it's probably it's yet another project I probably won't get around to, but, um, uh, yeah, if anyone wants to play with a badge, let me know. I mean, yeah, that was a, that was a questionable choice. Uh, but I had a good time and I didn't die, so, you know
Vyrus: and you're talking to.
Ben: yeah. But they may be prerecord the talks. So the talk was available. This was super weird. Right. So, uh, I had to prerecord the talk in like April or something. Um, and so, uh, by the time I actually gave the talk, uh, they released all of the prerecorded talks, uh, at the, as soon as the con started.
And my talk wasn't until Saturday. So I had been walking around for two days, getting messages on my phone, like, Hey man, saw your talk online. It was a good talk like before, like two days of that before I ever gave the talk. And then also, like I had kind of forgotten what was like, what was in the talk to some extent.
So what I did before I gave my talk is I watched a video of myself giving the talk like twice in a row and then went in and went on stage and tried to repeat myself from April. It was, it was a
Joe Fitz: of those.
Vyrus: There's been a lot of conversation on the review board about how like, like usually like, even, even under the best intentions invest scenarios, right? Like, you know, it's, it's like buying anything, right? Like it always looks better on the box. You know what I mean? Like people get nervous and people freak out.
Like, I'm not saying that like, people are intentionally, you know, diluting the quality or whatever, but like, it just happens. Right? Like there's more stuff and it needs space. Like usually the after, you know, interactions are amazing. That's the thing that like, no one can duplicate. Right. But generally speaking, like the talk description is always slightly better than like what most audiences take away.
From the attendance experience. And this year it's like totally backwards, right. Because everybody was super paranoid that like, oh my God, it's just the video. I don't have the excuse of like, you know, I'm going to go onstage and someone's going to hand me a drink and I get to have Ty, you know, conversations with people afterward.
And it's really just about the after interaction. It was like, no, your whole experience is focused on this one, one hour video that you have to create by yourself and people like, I mean, I guess the anxiety was good for people aside from the probable mental stress is like, the quality is really good.
Ben: it was, it was a heavy lift, but I actually kinda liked to doing it way ahead of time because it was done. Like
Vyrus: show up and like enjoy yourself.
Ben: showed up, I enjoyed myself. I went on stage. I ha I had for the first time in my life, I had like no stage fright because in my head it was like, I have already done this. This is already done.
Right. They didn't. And they didn't even record the live talk cause they already had, uh, the, the prerecorded one for me.
Um, so.
Vyrus: that handled all the pre-recording stuff like, oh my God. Perhaps with that team, look, they really, because like when the plan with like the reason everybody had to prerecord the talk is because, uh, talks were already done and in the can, before it was decided that definitely wasn't going to be fully remoting. the whole thing was like, kind of right up until the last moment. It was like, we're just going to go remote again. We don't know if we're going to be hybrid. We don't know if we're going to be hyper. We don't know if we're going to be hybrid. And like, even when the decision was made for folks to go hybrid, like it took a while for the message to kind of reverberate through the planning folks and like a common experience was, well, I guess we're going remote because I'm seeing us email speakers saying if you're coming internationally, we'll deal with some extra hoops because it's the pandemic.
And so like, that's the thing. And it was like, well, I guess we're doing like hybrid then. Cause when people didn't know for like ever
Ben: Yeah, well, and actually in, in parts of the Def con website actually had the wrong information about for awhile. Like there were delays in updates. So like even after it was like, yeah, we're going to, you know, we're doing this partially in person, there was a part of the website that was like, it's still canceled.
It's going to be a hundred percent remote.
Vyrus: yeah. That was hard. Cause it was like, you know, you have your regular like, like there's still a sock. Right. But none of the usual suspects show up because pandemic, well, that's not true. Like actually a bunch of the usual suspect goons actually did show up because they were like, if
Ben: Almost all the goons were there. Yeah.
Vyrus: yeah.
Cause they were like, if I'm going to die for anything, it's going to be like, you know, it's going to be like for my fellow Boone, so to speak, like they all kind of shut up to keep each other safe.
Ben: Well I think, I think basically goons show up, ready to accept some percentage risk of death every year.
Vyrus: It's not wrong, not wrong, but in this case it was more like, you know, it's been a pleasure playing with you, gentlemen. You know what I mean? It was a lot of that. Um, or people like, I don't know, whatever gender neutral of gentlemen is and, uh, it, you know, so that gentle folk, there we go. But so like, that was a thing.
Right. But then it was the whole problem of like, well, like safe mode really did pull together a kind of completely, not completely different, but like completely different, like organically, very tight knit group of people running the discord. Right? Like that was like a whole thing. So now you have these like kind of separate, like somewhat well-oiled machines that now need to sync.
And like, that was why sometimes there's like website flippage and like weirdness. But I dunno, man. It was, we all expected a bigger train wreck than we got, in my opinion,
Ben: Yeah, I, um, yeah, for real, um, although one of the, one of the funniest parts of the whole thing was, uh, you know, how they do the presentation at the end of like the, um, you know, mark goes up and talks about, uh, all the safety incidents
Vyrus: like the one, the one asshole who is like, what mask.
Ben: Yeah, well, no, all the incidents of like harassment and everything that happened at the event.
Um, uh, so he gave a really, you know, his, his, uh, uh, closing ceremony talk was really, uh, you know, he's always super funny, but like, um, it was really funny this year, cause like all the numbers of, uh, uh, you know, bad incidents where like all time lows, you know, Which is probably related to the, uh, uh, almost total lack of party.
Vyrus: Or just lack of party induced looky-loos
Ben: Yeah. Yeah. But, uh, but like all the people I actually talked to you, I mean, it was a Def con you just walk in and talk to strangers again, you know, and everyone I talked to was like super happy to talk about their like esoteric technical interests. Uh, there weren't any people who were like, oh, I just heard this was cool.
And then showed up and wanted to see what it was like, those people stayed home, you know? So like,
Vyrus: the point.
Ben: yeah, every, everyone was there who was there, like had some kind of mission. Um, and I found it to be like really energizing.
Vyrus: Clearly you've stumbled upon the key of the future is like whenever the quote unquote actual postpone pandemic occurs, the secret is we need to ramp up the messaging on Def con plague. Like quite clearly, the social message we need to send is that Devcon plague is so much worse than any other con plague.
And if you value your life, don't go the Def con variate. Oh my God. Ah, t-shirts Def con Aztec. They've gotten buried.
Ben: I mean, it's kind of like Def con is canceled. Right. But, you know,
I think people have caught on to that one now.
Vyrus: fed some super happy to hear you're going to be at a tour camp with the fam man. I'm like, I never gotten any interviews with your kids before. So for the last event where they were there, and that was super cool because I had my oldest there and now I've got two.
Ben: Oh, so was it
Joe Fitz: Yeah.
Ben: actually that, that brings up something else I wanted to talk to you about? I totally forgot. So I think I want to say it was the last tour camp, uh, although that's like four years ago now, but the, the, the last tour camp was that the one where you did the, uh, the drone taco delivery.
Joe Fitz: Talk a little liberal. Yep,
Ben: You want to talk about that a little bit?
Like, how were you manually steer? You were manually controlling the drones.
Joe Fitz: Yeah. Yeah. You know, it was a drone with a rope tied to it and a basket hanging below and we'd deliver tacos. Um,
Ben: So people would like call you up from other campsites and be like, I want this many tacos and you would,
Joe Fitz: So our, our phone number was for taco. So you just dial four taco or taco
Max: Nice.
Joe Fitz: and uh, Yeah.
we just made some tacos. Kids loved it. let some kids fly the drone a little bit, but you know, not too much.
Cause you know, there's some precious cargo we were thinking about like, wait, like I don't want to be like the taco drone guy forever. So we're trying to think of like different things to do and like make it into a contest. One idea was like a pneumatic tube, brilliant burrito delivery.
Ben: oh, I love pneumatic.
Joe Fitz: But like that's not good enough.
Like we need, we need shitty pneumatic.
tube burrito delivery, like where you get a burrito at one end, get a plate at the other end.
Vyrus: So you need like robot arms that unwrapped a burrito on the way through and.
Ben: Well,
Joe Fitz: And then we're thinking like, how, how do we make, like, do we have likely the burrito catapult contest, like where you try and get the burrito on the plate from across the field. And, uh, yeah. So
Ben: will you put a burrito in it? You put a burrito in one end, but it actually taco salad.
Joe Fitz: there you go.
Ben: Yeah.
Vyrus: and then the wild animals descended upon the camp and ate everywhere.
Ben: Yes. Those deer are scary.
Max: Just
Ben: they
Max: to like a high pressure beam tube.
Ben: Oh man. So you were, uh, I remember I was talking to you about, uh, you like organizing the kids camp or whatever at a tour camp. Um,
Joe Fitz: it's a strong word?
Ben: well, yeah, I mean, so you were talking about how you ended up kind of running, uh, uh, besides Portland cause everyone else sort of deferred. Right. There's also how I ended up running stuff.
It's like the only way I end up running stuff, for example, this podcast. Yeah. It's a,
Vyrus: something, something, this is how besides exists.
Ben: so what time of year is besides Portland? Usually?
Joe Fitz: It's usually October late October.
Ben: Okay.
Joe Fitz: yeah, So we took the year off. Everyone was delighted that that was fine with everyone. So.
Ben: yeah. Yeah. Well, hopefully it will be back next year and, uh, hopefully, uh, a tour camp won't get pushed back again cause that's, uh, that's always a highlight.
Joe Fitz: Cool.
Ben: so like at the kid's camp last time you had those like RGB, um, lights in the buckets.
Joe Fitz: Yeah. I will say that was, that was a control edge project. the internet buckets.
Ben: Yeah. Those were super cool. Those control age was someone there with a light table too with the sand.
Joe Fitz: Oh yeah. That's my sandbox
Ben: Uh, I totally wanted to build one of those. Uh, so the idea there is, you've got a, what is it called? Kinetic sand.
Joe Fitz: Yeah. So I put connect and sand in there. So sand plus, uh, some silicone material that makes it a little clumpy without getting moldy. Um, and, uh, yeah, you,
it just makes the handle more comfy and then you have a projector on the top and a connect that does depth mapping and it projects, mountains and rivers and stuff.
I actually rebuilt it since control. Since, since that last time I replaced it with an Nvidia Jetson board. Um, so it has a GPU, so it does all the, uh, rain simulation as well.
Ben: holy
Max: Oh, that's all.
Ben: Oh man. I can't wait to see that, but it's so, it's so cool. Cause like you go in the sandbox and you like make a little pile and it figures out because the pile is. That, should be a mountain. And then the projector above it, like project a mountain there. And if you make a, you know, like a, like a lower area, it fills it in with water, you know, it's um, and I just love that thing.
Vyrus: I feel like if we can figure out how to make the grains of sand, like magnetic and make the bottom plate, like, you know, something like, I feel like we're Columbus, right? We're close to like the, the Saifai like
Max: a tactile
Vyrus: like physical projected, like model thing.
Ben: Oh, man. It has been such a pleasure to talk to you, Joe. Uh, I thank you so much for coming on here and, uh, sharing your wisdom. Um,
Joe Fitz: thanks for having me. Good talking to all of you.
Ben: yes, it's, uh, can't wait to, uh, to do this in person.
Mitchell: Yeah. Uh, I just wanted to say, like, you've probably been a pretty big force multiplier personally, and like hardware security in general. So I don't props for that.
Max: Absolutely. I think that you've definitely helped, helped me guide my path throughout the industry and we've, uh,
so.
Ben: I mean, you know, I've, you know, I've just texted you stupid questions all year,
Joe Fitz: every
Ben: all, you've been very considerate about, uh, explaining how to do basic stuff to me, uh, over over many years, including this one. So
Joe Fitz: every time I open up a cheap device and I find a UART console and I don't get a root shell, I always worry that.
it's my own fault.
Max: That's pretty
Ben: Yeah,
Max: good. Bosses syndrome is real,
Ben: that's true.
Vyrus: actually not done. I have a whole bunch of questions for you now. Now the know about a certain of.
Joe Fitz: Uh, we should talk then. Cool.
Ben: well, uh, yes, that's been Joe Fitzpatrick. Uh, definitely check out his site, securing hardware.com, uh, and you know, many of his other projects, I'm sure. Oh, the T guard on, um, Tigard Tigard on crowd supply and look out for the Airbus, which are you putting the Airbus out on crowd supply also?
Joe Fitz: most likely, but that's like a year from now, maybe.
Ben: Yeah. Well, at the rate we get these podcasts edited, uh, that might be a
Joe Fitz: Yeah.
Ben: current event.
Joe Fitz: Any day now,
Ben: Yeah. So, so, uh, look for the Airbus on crowd supply. Yeah. I need to know exactly. All right,
Max: My project
Ben: man. How do we end these things
Vyrus: There's that catchphrase thing.
Ben: now? More than ever.
Max: more more than ever. Hacking is not just the crime.
Mitchell: It's a survival tray.