If your not a pilot this is probably not meaningful for you.
My FAA medical lapsed in 2007 and and I have not flown since. Long time followers of my blog will remember that 3 or so years ago I had a post about Sleep Apnea and getting a CPAP machine. This significantly complicates getting a flight physical. Before I figured out the sleep apnea there were some other minor medical issues due to the poor sleep that I dealt with that also complicate the flight physical. The FAA flight physical does not have a recency or last x years stipulation, the form says "have you ever".... and in today's realm of pervasive computer databases your going to get busted if you are not 100% honest about EVERYTHING!. Every yes on the form must be documented and dealt with.
The FAA Medical process as it is currently construed has a really nasty catch 22, One can fly light sport aircraft (2 seats, day vfr, gross less than 1320 lbs) with a drivers license instead of a medical, as long as you have never been denied a medical. So going to the FAA to get a 3rd class medical has risk, if you try and fail then you loose the LSA option, If you never try and never fail you keep the LSA option.
So the only way to really combat this is to gather all the necessary paperwork, all the tests reports, documentation records etc... and go individually do equivalents of all the FAA medical tests, vision, etc...so you know what the answers will be before you actually start the process.
How do you know what tests and documents will be required? You need expert advice, and while there are large companies charging thousands of $ for this help I found them to be unresponsive and bureaucratic.
(Yes I paid the initial FEE to a large aviation medical consulting co and got ZERO value for the large $) and then I found Dr Bruce Chien participating on the Pilots of America medical forum.
I contacted him directly and I could not be more pleased with the whole process. He is a AME that specializes in difficult medicals. He worked with me to identify possible issues, then gather and document these in a way that would be acceptable to the FAA. Finishing this process required that I fly to his office in Il, in the end even including airfare, hotel, rental car etc.... it was less than half the cost of going through one of the big aviation medical consulting firms.
So I now have a brand new 3rd class medical, and I'm really looking forward to flying again....
Reasonable people adapt themselves to the world. Unreasonable people attempt to adapt the world to themselves. All progress, therefore, depends on unreasonable people. - George Bernard Shaw.
Tuesday, October 29, 2013
Monday, August 05, 2013
Space and futures....
It looks like Armadillo is now dormant or dead.
Based on some hints I've received, I've suspected that this was coming for awhile...
The suborbital business never made any sense to me, its hard and other than artificially created markets (cruiser program) and thrill seekers I see no fundamental business case to be had at a price that is remotely appropriate to the level of effort.
Its been 10 years since space ship one and I see no one close to having a profitable business
launching suborbital passengers or payloads.
I've stared longingly at this business, I'd really like to do something here, but no matter how hard I look it has never made sense as a business...
There is remote possibility that someone might come up with a nanosat launch business that makes sense.(I've even written a somewhat optimistic business plan toward that end), but once you add it the difficulties of testing and regulatory compliance the whole thing is really really hard. For the time being it looks like the only real progress to be made will be done by large well funded organizations like SpaceX and possibly Blue Origin, but even Blue is subject to the whims of Bezeo's attention, he may find it more fun to play with his new newspaper...(He jsut spend 250M of his own $ on the Washington post)
I'd like to see someone try to build an expendable launcher where the primary aim is automated manufacturing and really low cost. I think that even if Spacex gets its 1st stage re-usability right it would be possible to beat them on cost, they currently have close to 3K employees, at reasonable aerospace wages at 33 flights a year that is 10M a launch just for wages..... Too bad Beal failed, I think they were really on the right track for low cost space access...
Dam stubborn gravity well....
Based on some hints I've received, I've suspected that this was coming for awhile...
The suborbital business never made any sense to me, its hard and other than artificially created markets (cruiser program) and thrill seekers I see no fundamental business case to be had at a price that is remotely appropriate to the level of effort.
Its been 10 years since space ship one and I see no one close to having a profitable business
launching suborbital passengers or payloads.
I've stared longingly at this business, I'd really like to do something here, but no matter how hard I look it has never made sense as a business...
There is remote possibility that someone might come up with a nanosat launch business that makes sense.(I've even written a somewhat optimistic business plan toward that end), but once you add it the difficulties of testing and regulatory compliance the whole thing is really really hard. For the time being it looks like the only real progress to be made will be done by large well funded organizations like SpaceX and possibly Blue Origin, but even Blue is subject to the whims of Bezeo's attention, he may find it more fun to play with his new newspaper...(He jsut spend 250M of his own $ on the Washington post)
I'd like to see someone try to build an expendable launcher where the primary aim is automated manufacturing and really low cost. I think that even if Spacex gets its 1st stage re-usability right it would be possible to beat them on cost, they currently have close to 3K employees, at reasonable aerospace wages at 33 flights a year that is 10M a launch just for wages..... Too bad Beal failed, I think they were really on the right track for low cost space access...
Dam stubborn gravity well....
Monday, June 17, 2013
LENR I don't think I'm crazy...
The AVC is over its time to talk a little bit about my LENR project.
24 years ago or so Pons and Fleischmann made a big splash with claims of Cold Fusion, ie using simple electrochemical cells generating excess heat beyond what could be explained by chemical energy.
This caused quite a controversy and after a few years the furor died down and it seemed like Cold Fusion was largely discredited. At the time my Dad was a commodities trader and I bought Pd futures and made a few hundred dollars by riding it up and most of the way back down.
Every once in awhile since then I'd hear of some experiment or another and then it would go dark. About 6 mo's ago a non-technical someone asked me to review some stuff in this field for technical validity as they were contemplating an investment. I expected to report back that t was a fraud or junk science. What I discovered surprised me.
You may believe me or not but this is what I now believe...
The Pons and Fleischmann effect is real. (here after called PFE)
To this day we still do not understand what is going on, replicating the PFE requires Pd of specific structure, and unknown contaminates. A number of cases have been shown where a specific Pd rod that works in one lab will work in a different lab with different apparatus. Just plain unscreend Pd works about 5% of the time. Pre screening the Pd for high D-Pd loading ratios without Pd swelling can reduce this ratio to about 33%, ie if a Pd rod will absorb > 0.95 D for each Pd atom without swelling and cracking the odds of it demonstrating the PFE is 30% or better.
Whatever is happening it does not generate neutrons expected of traditional hot fusion., and is probably not traditional fusion. There have been a large number of theories presented, none of them presently fit all of the facts, so we effectively have no theory. Over 60K peer reviewed papers have been written about High temp superconductivity and we have no theory there either, so if science was functioning properly this lack of theory should not be a total block to progress.
I think one of the pathological failures was calling it Cold fusion, because it's probably not fusion in the traditional sense. Much more likely to be some path following the weak force rather than the strong force...Also there is now a growing body of evidence that whatever is going on can happen in regular water and ni, so some of the early "dead" experiments that were used to prove an instrumentation error were later found to be lower grade PFE reactions on their own.
Another body of evidence also shows that the Pd needs to have some "Stuff" plated on it to work really well, P+F were a tiny bit sloppy about contamination, so when it came time to replicate the more meticulous the lab the less chance they had of replication. Its also been shown that most of the successful replications from the time of early P+F all came from the same batch of Pd rod all contaminated with a few ppm of Rhodium and other trace elements.
Some of the experiments show that it takes 100hrs+ to load a Pd rod enough for it to turn on... again it does not help replication....
My Friend Dave W, says its sort of like the story: I've been told that if I put a bent piece of shiny metal and small bit of wire on the end of a string and dip it in water I can pull up free food. I've been doing in my lab with sterilized string and distilled water for three weeks and still no food....
We clearly don't know what we are missing...
An ideal energy source that runs off of water is too good to be true and it attracted a bunch of scammers eager to make a buck. These scammers did not do the real scientists any favors. I'm still not convinced that the most notorious of the lot Rossi is not a scammer/huckster.
I am convinced that the Navy SPAWAR lab, SRI, briillion, Ed Storms, Toyota and others have done very careful science, addressing every issue brought up in opposition to their results and have proven to me beyond a doubt that there is a PFE and there is science here we don't understand.
So all the scientists with a desire to be accepted by other scientists have worked very hard doing careful Calorimetry running experiments lasting 100's of hours proving that the result can't be chemical.
A properly done long term calorimitry experiment is the gold standard of proving its not a chemical effect....
If your experiment takes 100hrs to complete then its really hard to do fast cyclical testing.
Each sample you try is a major commitment of time and care.
My idea is this, accept the PFE is real. Try lots and lots of samples and screen for it.
Don't wiat 100hrs to toss a sample that is not working. Toward that end I've built a tiny little stainless reaction chamber that I can heat and pressurize with H or D and stimulate the sample with RF or electrical pulses ... (the system is 50 ohms up to and back out of the sample chamber.) I've got an fast reading 15msec IR pyrometer with a tiny spot size that I can use to measure the instantaneous temperature of the sample, so if some stimulation causes the sample to "go" I don't have to wait hours to know what I just did.
(The IR pyrometer looks through a sapphire window rated for the temps and pressures I expect)
I want to try dozens of samples a day, try 1000's of possible stimulation....
I also have 4 wire restive measurement of the sample and a large pancake style GM tube to detect any radiation form the experiment.
The wright bothers did not build a plane first, first they built a wind tunnel.
I'm building a LENR wind tunnel, the samples will be tiny and useless, but if they generate excess power above the chambers elevated temperature I'll know quickly and hope to be able to optimize to maximize the effect. in the next day or two I'll organize my pictures of the reactor and test equipment and post it all here...
I'm on my second revision of the reactor, the first revision was a bit too clunky to assemble and disassemble, the current version seems much easier.
In this same theme I'm going to the ICCF-18 in July...
24 years ago or so Pons and Fleischmann made a big splash with claims of Cold Fusion, ie using simple electrochemical cells generating excess heat beyond what could be explained by chemical energy.
This caused quite a controversy and after a few years the furor died down and it seemed like Cold Fusion was largely discredited. At the time my Dad was a commodities trader and I bought Pd futures and made a few hundred dollars by riding it up and most of the way back down.
You may believe me or not but this is what I now believe...
The Pons and Fleischmann effect is real. (here after called PFE)
To this day we still do not understand what is going on, replicating the PFE requires Pd of specific structure, and unknown contaminates. A number of cases have been shown where a specific Pd rod that works in one lab will work in a different lab with different apparatus. Just plain unscreend Pd works about 5% of the time. Pre screening the Pd for high D-Pd loading ratios without Pd swelling can reduce this ratio to about 33%, ie if a Pd rod will absorb > 0.95 D for each Pd atom without swelling and cracking the odds of it demonstrating the PFE is 30% or better.
Whatever is happening it does not generate neutrons expected of traditional hot fusion., and is probably not traditional fusion. There have been a large number of theories presented, none of them presently fit all of the facts, so we effectively have no theory. Over 60K peer reviewed papers have been written about High temp superconductivity and we have no theory there either, so if science was functioning properly this lack of theory should not be a total block to progress.
I think one of the pathological failures was calling it Cold fusion, because it's probably not fusion in the traditional sense. Much more likely to be some path following the weak force rather than the strong force...Also there is now a growing body of evidence that whatever is going on can happen in regular water and ni, so some of the early "dead" experiments that were used to prove an instrumentation error were later found to be lower grade PFE reactions on their own.
Another body of evidence also shows that the Pd needs to have some "Stuff" plated on it to work really well, P+F were a tiny bit sloppy about contamination, so when it came time to replicate the more meticulous the lab the less chance they had of replication. Its also been shown that most of the successful replications from the time of early P+F all came from the same batch of Pd rod all contaminated with a few ppm of Rhodium and other trace elements.
Some of the experiments show that it takes 100hrs+ to load a Pd rod enough for it to turn on... again it does not help replication....
My Friend Dave W, says its sort of like the story: I've been told that if I put a bent piece of shiny metal and small bit of wire on the end of a string and dip it in water I can pull up free food. I've been doing in my lab with sterilized string and distilled water for three weeks and still no food....
We clearly don't know what we are missing...
An ideal energy source that runs off of water is too good to be true and it attracted a bunch of scammers eager to make a buck. These scammers did not do the real scientists any favors. I'm still not convinced that the most notorious of the lot Rossi is not a scammer/huckster.
I am convinced that the Navy SPAWAR lab, SRI, briillion, Ed Storms, Toyota and others have done very careful science, addressing every issue brought up in opposition to their results and have proven to me beyond a doubt that there is a PFE and there is science here we don't understand.
So all the scientists with a desire to be accepted by other scientists have worked very hard doing careful Calorimetry running experiments lasting 100's of hours proving that the result can't be chemical.
A properly done long term calorimitry experiment is the gold standard of proving its not a chemical effect....
If your experiment takes 100hrs to complete then its really hard to do fast cyclical testing.
Each sample you try is a major commitment of time and care.
My idea is this, accept the PFE is real. Try lots and lots of samples and screen for it.
Don't wiat 100hrs to toss a sample that is not working. Toward that end I've built a tiny little stainless reaction chamber that I can heat and pressurize with H or D and stimulate the sample with RF or electrical pulses ... (the system is 50 ohms up to and back out of the sample chamber.) I've got an fast reading 15msec IR pyrometer with a tiny spot size that I can use to measure the instantaneous temperature of the sample, so if some stimulation causes the sample to "go" I don't have to wait hours to know what I just did.
(The IR pyrometer looks through a sapphire window rated for the temps and pressures I expect)
I want to try dozens of samples a day, try 1000's of possible stimulation....
I also have 4 wire restive measurement of the sample and a large pancake style GM tube to detect any radiation form the experiment.
A picture inside the reactor the depression holds an alumina plate and the ceramic feed through on both sides are 50 ohms up to the wall, and with 0.016 thick wire the plate is the right thickness for 50 ohms.
Both sides of the reactor with the sapphire sample window visible.
The reactor inside its heater jacket with thermocouples attached, the PID controller holds a constant temperature. Design operating point is 300 PSI and 600C with SF of 3.0. You can see the IR pyrometer lying on its side next to the reactor, I still need to fabricate an adjustable support to make it peer through the window.
I'm building a LENR wind tunnel, the samples will be tiny and useless, but if they generate excess power above the chambers elevated temperature I'll know quickly and hope to be able to optimize to maximize the effect. in the next day or two I'll organize my pictures of the reactor and test equipment and post it all here...
I'm on my second revision of the reactor, the first revision was a bit too clunky to assemble and disassemble, the current version seems much easier.
In this same theme I'm going to the ICCF-18 in July...
Saturday, June 15, 2013
AVC Part 2
The world does not want me to write this post....
The car makes it through the hoop but misses the ramp... slightly to the left.... I'm faster than Savage solder, but they hit the ramp... after the first heat the score is Savage Solder 440 NetBurner (me) 412.. so I was 22 seconds faster... but missed the 50pt hoop bonus.
I hurry back and look at the small car code.... I see where I commented out too much and that my {} in the state machine did not line up as expected, id turned off the IMU/gyro processing. It is with high hopes I go back to run heat two of the Cheapshot small car.... This time the car starts right up, turns hard just like its big brother and flips over on the start ramp.... I'm now zero for 2 on this vehicle.
Time to adjust the course to try and hit the ramp and run the big car again....I move the way points over about 3 ft to hit the ramp and we run heat two.... It lurches off the start gets going solidly.... just three feet before the hoop it swerves out of the way just missing the hoop.... when it gets to the ramp it goes way wide and actually rubs on the curb..... I moved the way points the wrong way.... it scrapes along the curb recovers and finishes the race..... I'm now solidly in first place.
(After heat 2 Photo by Ben Brockert)
I'd worked on it for about an hour and forgot to open a new tab when navigating away to find a link and blogger lost it.....
On the day of the event I arrive at the venue at around 7:45, Unloaded my stuff and setup on the provided worktables under the tent. They open the course for testing at 8am and I slowly walk the doping class car around the course gathering way point data. For the first turn I decide to go waaay long so as to try and avoid some of the traffic. In the videos you can see the car goes way past the first turn this is intentional.
My first heat is with Cheapshot the little car. Its heat two and a rush to be ready. I strip out the RC code and setup so the system arms on reset..... I test this in the pits by pressing the go button and it seems to work.
I sit at the line its time to go I hit the button and nothing happens..... I then hit rest and wait out the 20 seconds of gyro zero and other things then the car finally takes off just after the first place car finishes.....
The car goes out about 10 feet and turns in a large never ending never varying circle round and round...
Heat 2 round 0 Zero....
So I have 6 heats or so to get the big car ready.... I revert the code to what it was for testign last night. Rather than hurrying to strip otu the RC arming code I just comment out the one line of processing received chars from the DSM2 receiver and add a serial port command to arm the car....
I press the button and the car turns hard right smacking into other cars... then it straightens out and runs the course solidly. (Here is the heat one video..)
Video by Ben Brockert
The car makes it through the hoop but misses the ramp... slightly to the left.... I'm faster than Savage solder, but they hit the ramp... after the first heat the score is Savage Solder 440 NetBurner (me) 412.. so I was 22 seconds faster... but missed the 50pt hoop bonus.
I hurry back and look at the small car code.... I see where I commented out too much and that my {} in the state machine did not line up as expected, id turned off the IMU/gyro processing. It is with high hopes I go back to run heat two of the Cheapshot small car.... This time the car starts right up, turns hard just like its big brother and flips over on the start ramp.... I'm now zero for 2 on this vehicle.
Time to adjust the course to try and hit the ramp and run the big car again....I move the way points over about 3 ft to hit the ramp and we run heat two.... It lurches off the start gets going solidly.... just three feet before the hoop it swerves out of the way just missing the hoop.... when it gets to the ramp it goes way wide and actually rubs on the curb..... I moved the way points the wrong way.... it scrapes along the curb recovers and finishes the race..... I'm now solidly in first place.
(After heat 2 Photo by Ben Brockert)
They now take a break and let everyone practice for an hour before the next heats. Realistically the small car has no chance of even placing so I focus on the getting the big car to hit the ramp. I go out on the course and carefully measure the center of the ramp.....
I do a quick review of the code for the small car and realize that steering code has a static variable in it to measure the rate of turn... between loops and since most of my testing had the course starting to the north this would not be apparent.. I change the code so it ignores this for the first pass through navigation and prepare to try again.... This time the small car ends up in the fence... I'm zero for 3 very frustrating..
I'd planned on going faster than I was going. In testing I ran the straightaways at 3X the speed and 50% faster on the corners, but for the last run I had such a lead that if I just finished I would win. No incentive to go faster... It was still fast enough to jump the finish line:
So I left the car just as it was and the third heat looked just like the first two....it went straight at the hoop and the ramp and dodged away just missing at the last possible minute.
It did not matter I was not fastest, I was most consistent I'd won.
I think I had 1131 points and the 2nd place car (the team that won last year 0x27) had 571 points.
That is a pretty solid win. If Id just focused on the one car I could have fixed a few bugs and gotten another 250 pts...(2 hoops and 3 ramps).
So what did I learn? Again I relearned the don't try and do too much... one entry, not three. I Was lucky that my starting out weirdness did not cause a crash and if the 2nd/3rd place cars had been consistent I would have taken third.
The way I tested I drove the car around an area laying out a track... then placed way points on the map. Since the map faced north all of my testing had me starting to the north.... something about my starting code worked on a heading of 0... but not at a heading of ~270 I thought I was testing like I'd race... alas that was not the case.
The BD960 trimble GPS and Omnistar service was flawless the entire time I used it. Sitting in the tent at the race it was reporting an estimated error of between 5 and 15cm. Without this level of accuracy winning the race would have required a lot more work. Its pretty amazing when you realize that its measuring better than 20cm at 20hz.
The GPS track was flawless. better than 20cm accuracy at 20Hz, so why did I miss the hoop twice and the ramp every time? After my midnight suspension repair the car pulled to the right. The code took care of this with an integral steering term in the PID that turned heading into steer direction. As all the turns were in the same direction and I did not want windup I zeroed the I term at each waypoint switch.... so since I put waypoints right before the hoop and the ramp.... right before the hoop and the ramp the I term was zeroed and the car swerved to the right until the error built up again.... I should have checked the heading change and not zeroed the I term for changes less than some value. I should have figured this out in testing,....
In my testing I'd set the hoop in the track and made sure the car was consistent. I never tested hitting a spot the was pre-configured, only in being consistent run after run... (You can see the in the video from my previous post) The track was very consistant, but it wasn't wher eI wanted it to be.
Last year I handicaped myself by not using the good GPS. This year I wanted to prove a point that the lowest cost NetBurner device was powerful, so I gave myself a deliberate handicap by choosing our lowest powered CPU. In testing I recorded telemetry and got data from the devices. During the actual race I had no on board data recording. If I had I would have easily seen both the the start up and ramp missing problems in the data in a way that I could fix them... regardless the car won with an SBL2E cpu, a network attached CPU for less than $15.
I've now won with a car so next year I'm going to finally enter a flying machine. Id prefer to enter a plane, but if the rules don't change they heavily favor a VTVL vehicle. So I'll probably pull my autonomous helicopter out of mothballs and use a new modern CPU. For fun here is the best piece of video my autonomous helicopter ever took:
I do a quick review of the code for the small car and realize that steering code has a static variable in it to measure the rate of turn... between loops and since most of my testing had the course starting to the north this would not be apparent.. I change the code so it ignores this for the first pass through navigation and prepare to try again.... This time the small car ends up in the fence... I'm zero for 3 very frustrating..
I'd planned on going faster than I was going. In testing I ran the straightaways at 3X the speed and 50% faster on the corners, but for the last run I had such a lead that if I just finished I would win. No incentive to go faster... It was still fast enough to jump the finish line:
(picture by maushammer)
It did not matter I was not fastest, I was most consistent I'd won.
I think I had 1131 points and the 2nd place car (the team that won last year 0x27) had 571 points.
That is a pretty solid win. If Id just focused on the one car I could have fixed a few bugs and gotten another 250 pts...(2 hoops and 3 ramps).
So what did I learn? Again I relearned the don't try and do too much... one entry, not three. I Was lucky that my starting out weirdness did not cause a crash and if the 2nd/3rd place cars had been consistent I would have taken third.
The way I tested I drove the car around an area laying out a track... then placed way points on the map. Since the map faced north all of my testing had me starting to the north.... something about my starting code worked on a heading of 0... but not at a heading of ~270 I thought I was testing like I'd race... alas that was not the case.
The BD960 trimble GPS and Omnistar service was flawless the entire time I used it. Sitting in the tent at the race it was reporting an estimated error of between 5 and 15cm. Without this level of accuracy winning the race would have required a lot more work. Its pretty amazing when you realize that its measuring better than 20cm at 20hz.
The GPS track was flawless. better than 20cm accuracy at 20Hz, so why did I miss the hoop twice and the ramp every time? After my midnight suspension repair the car pulled to the right. The code took care of this with an integral steering term in the PID that turned heading into steer direction. As all the turns were in the same direction and I did not want windup I zeroed the I term at each waypoint switch.... so since I put waypoints right before the hoop and the ramp.... right before the hoop and the ramp the I term was zeroed and the car swerved to the right until the error built up again.... I should have checked the heading change and not zeroed the I term for changes less than some value. I should have figured this out in testing,....
In my testing I'd set the hoop in the track and made sure the car was consistent. I never tested hitting a spot the was pre-configured, only in being consistent run after run... (You can see the in the video from my previous post) The track was very consistant, but it wasn't wher eI wanted it to be.
Last year I handicaped myself by not using the good GPS. This year I wanted to prove a point that the lowest cost NetBurner device was powerful, so I gave myself a deliberate handicap by choosing our lowest powered CPU. In testing I recorded telemetry and got data from the devices. During the actual race I had no on board data recording. If I had I would have easily seen both the the start up and ramp missing problems in the data in a way that I could fix them... regardless the car won with an SBL2E cpu, a network attached CPU for less than $15.
I've now won with a car so next year I'm going to finally enter a flying machine. Id prefer to enter a plane, but if the rules don't change they heavily favor a VTVL vehicle. So I'll probably pull my autonomous helicopter out of mothballs and use a new modern CPU. For fun here is the best piece of video my autonomous helicopter ever took:
Lastly I get requests for the code on these projects.... since I write code for a living for NetBurner I'm hesitant to post my hobby code because its not up to the standard of code that I would release in my professional capacity. So as long as you realize that this is hobby code written in a hurry while suffering from sleep deprivation you can have at it..
.
.
Here is the code for the car.. as a a few days before the AVC...
and
here is the QT code for taking logs and placing waypoints... again a few days before the AVC....
I just returned from the SparkFun 2013 AVC. It was another interesting experience.
This is my second year competing and I'd hoped I'd learned something. (See 2012 report).
At one level this year was a success I won the doping class rover with over 1100 pts, 2nd place was at 571 pts. I set out to win and I did, beyond that it was much more painful than I expected it to be.
The format of this years contest differed from last year and my LASER range finder approach would have been problematic. In any case the Trimble BD960 L2 Omnistar GPS I had from the lunar lander contest was the perfect instrument to make this contest seem easy....
Last year I ran a red cat racing 1/5 scale buggy and so did last years winner, team 0x27.
This year I decided to bring three vehicles to the event:
Insanity is doing the same thing over and over and expecting a different outcome. Last year I tried to do both a plane and car and failed from having too much on my plate this year I again tried to do too much... I got much farther with the plane than last year, it was very close to ready. I could have successfully done the minimum necessary to complete the task. The ball drop, and autoland were not ready and hence it could not win, or even place.
So lets back up a couple of months....I'm a pilot from way back. (my Dad ran an Alaskan bush airline and I've been flying since I was a teenager.) And I love airplanes, my heart was reallyu into building an airplne for the contest and toward that end I developed a custom autopliot board using the same sensors as the 3D robotics APM and a NetBurner MOD5213 for the CPU. I put together a nice clean installation in a park zone Extra 300 . One pof the stretch goals was to develop some hardware and software that might eventually become a product.
Picture of the Autopilot in the canopy of the test plane.
I spent about three months working on the code and hardware to make all of this clean and tight in the end I ran out of time. Along the way I tried to use a Q2 Robotics controller I got from kickstarter and getting that to work correctly ate a whole month and was finally abandoned after I re-cpu-ed it and ran out of time.
I'd hoed to use the controller for both the plane and the cars, in developing an autonomous vehicle its really useful to be able to get real time feedback and to have lots and lots of mode switches. I also liked the four pots allowing one to tune 4 different PID tuning parameters in real time.
Here is a picture of my final version:
In preparing the cars I started by bolting the high end BD960 GPS to last years car and getting it all working.
I also chose a new chassis, a Traxss XO-1 the fastest production RC car on the planet. (I was playing to win)
About this time I decided that since 99% of the work was in the software and since I wanted to run both a leess than $350 class car and a doping class car I chose the same CPU for both. TheCheapShot got the Netburner SBL2E Break out board for the CPU. Since this CPU uses a different branch of the NetBurner RTOS environment it meant that the car code was an almost 100% complete re-write. I used the same CPU in a different package for the eventual Doping class car. This CPU is limited and is made for really low cost network attached device so it presented some challenges:
Here are pictures of the three
Last years CPU had SDCARD and 64Mb of RAM so logging data was not an issue.
This years CPU did not have either a SD Card or enough RAM to log all data.
So I started using a SparkFun data logger to record data, but it soon became apparent that at 115K baud it just did not keep up. So I added an Xbee to radio telemetry to the computer and stay under the $350 cost target. I eventually ended up with two schemes one using the Xbee on the CheapShot car and one using a
a WIFI router on the Doping car.... as the tiny car was not big enough to carry the router.
So I developed two systems of telemetry encoding and recording... ugly...
On last years CPU I had a really nice system for setting and modifying parameters... without recompiling the code... I ran out of time to port this system to the new CPU so everything was compiled in as code constants.... I should have taken the time to fix this it would have been a win in time and effort, but hindsight is 20/20. In the heat of the process the pain never got bad enough to make me fix this.
I developed a weird trap bug about a week before the contest... I spent three or four days wringing custom diagnostic tools to find my bug only to find that it was an interrupt priority problem I had two interrupts with exactly the same priority and level and this causes a known bug in the CPU interrupt controller.....
So about two weeks before the contest I had the doping class car hardware finished and it was driving around in my driveway...
At this point it looks pretty good...
It was still missing:
This is my second year competing and I'd hoped I'd learned something. (See 2012 report).
At one level this year was a success I won the doping class rover with over 1100 pts, 2nd place was at 571 pts. I set out to win and I did, beyond that it was much more painful than I expected it to be.
The format of this years contest differed from last year and my LASER range finder approach would have been problematic. In any case the Trimble BD960 L2 Omnistar GPS I had from the lunar lander contest was the perfect instrument to make this contest seem easy....
Last year I ran a red cat racing 1/5 scale buggy and so did last years winner, team 0x27.
This year I decided to bring three vehicles to the event:
- A full on doping class vehicle done as well as I could with almost no restrictions.
- A low cost <$350.00 entry.
- A plane.
Insanity is doing the same thing over and over and expecting a different outcome. Last year I tried to do both a plane and car and failed from having too much on my plate this year I again tried to do too much... I got much farther with the plane than last year, it was very close to ready. I could have successfully done the minimum necessary to complete the task. The ball drop, and autoland were not ready and hence it could not win, or even place.
So lets back up a couple of months....I'm a pilot from way back. (my Dad ran an Alaskan bush airline and I've been flying since I was a teenager.) And I love airplanes, my heart was reallyu into building an airplne for the contest and toward that end I developed a custom autopliot board using the same sensors as the 3D robotics APM and a NetBurner MOD5213 for the CPU. I put together a nice clean installation in a park zone Extra 300 . One pof the stretch goals was to develop some hardware and software that might eventually become a product.
Picture of the Autopilot in the canopy of the test plane.
I spent about three months working on the code and hardware to make all of this clean and tight in the end I ran out of time. Along the way I tried to use a Q2 Robotics controller I got from kickstarter and getting that to work correctly ate a whole month and was finally abandoned after I re-cpu-ed it and ran out of time.
I'd hoed to use the controller for both the plane and the cars, in developing an autonomous vehicle its really useful to be able to get real time feedback and to have lots and lots of mode switches. I also liked the four pots allowing one to tune 4 different PID tuning parameters in real time.
Here is a picture of my final version:
In preparing the cars I started by bolting the high end BD960 GPS to last years car and getting it all working.
I also chose a new chassis, a Traxss XO-1 the fastest production RC car on the planet. (I was playing to win)
About this time I decided that since 99% of the work was in the software and since I wanted to run both a leess than $350 class car and a doping class car I chose the same CPU for both. TheCheapShot got the Netburner SBL2E Break out board for the CPU. Since this CPU uses a different branch of the NetBurner RTOS environment it meant that the car code was an almost 100% complete re-write. I used the same CPU in a different package for the eventual Doping class car. This CPU is limited and is made for really low cost network attached device so it presented some challenges:
Here are pictures of the three
The original low cost car.
The Low cost car as run....
The doping class car as it was run
Last years CPU had SDCARD and 64Mb of RAM so logging data was not an issue.
This years CPU did not have either a SD Card or enough RAM to log all data.
So I started using a SparkFun data logger to record data, but it soon became apparent that at 115K baud it just did not keep up. So I added an Xbee to radio telemetry to the computer and stay under the $350 cost target. I eventually ended up with two schemes one using the Xbee on the CheapShot car and one using a
a WIFI router on the Doping car.... as the tiny car was not big enough to carry the router.
So I developed two systems of telemetry encoding and recording... ugly...
On last years CPU I had a really nice system for setting and modifying parameters... without recompiling the code... I ran out of time to port this system to the new CPU so everything was compiled in as code constants.... I should have taken the time to fix this it would have been a win in time and effort, but hindsight is 20/20. In the heat of the process the pain never got bad enough to make me fix this.
I developed a weird trap bug about a week before the contest... I spent three or four days wringing custom diagnostic tools to find my bug only to find that it was an interrupt priority problem I had two interrupts with exactly the same priority and level and this causes a known bug in the CPU interrupt controller.....
So about two weeks before the contest I had the doping class car hardware finished and it was driving around in my driveway...
At this point it looks pretty good...
It was still missing:
- Start on a push button.
- Way to record and enter Waypoints other than by hand.
- IT was still dependent on the RC transmitter for throttle control.
At this point I started trying to run the code to the the less than $350 car.
I did not do much S/W development on this car as the low cost GPS was not working well enough to reliably stay in the driveway. I figured that I'd tune and tweak the settings for this car when I got to testing in a larger more open venue.
I did not do much S/W development on this car as the low cost GPS was not working well enough to reliably stay in the driveway. I figured that I'd tune and tweak the settings for this car when I got to testing in a larger more open venue.
Its now about a week before the contest and I go back to working on the plane.
(I also ran a 1/2 Marathon on Sunday the 2nd that I had signed up for before the AVC was on the schedule.)
I continued to have the plane will be ready fantasy until Wed morning when the canopy came off in flight and I totaled the primary air frame...
In the evenings when it was too dark to fly I started working on a mapping app to help me enter way points.
This was my first real QT app and while buggy and incomplete it worked reasonably well...
I left San Diego at noon on June 5th, headed for Boulder. After 1100 miles we arrived in boulder on the afternoon of the 6th. We went to the boulder reservoir and started driving around the parking lot making sure things were working well. At this point I had a way to manually enter waypoints overlayed on a telemetry track..and to run the car at fixed throttle without touching the RC transmitter....
As the parking lot started to fill up at 5pm it got too crowded to really test the cars... we went on to find our hotel.... I then spent the evening moving the electronics from the tiny fragile car to the less tiny less fragile car for the $350 class. After that was don I went to the parking lot across from the hotel to do some tuning and testing..... The GPS precision of the tiny car was so much poorer that the good gps tht it did nto drive very well, I needed to use the magnetic heading and turn down the response to the GPS errors.
So for the 350 car on Thursday I
- Moved the electronics
- Re-calibrated the magnetometer
- Spend several hours tweaking the steering and tracking gain.
At this point I have both cars driving around the parking lot with the RC receiver still hooked up and taking abort commands...
So back at the hotel ... I do the following...
The contest rules say no communication with the car, so I need to bring a car with no telemetry and no RC receiver to the event. With no telemetry how do I record the position of all the waypoints ?
so I add a system to record specific points into memory using the "Gear" switch on the RC transmitter.
I then go across the parking lot and test this.... The doping class car works just fine, the Low cost car with the pooer GPS and lower GPS rate drives like a drunken sailor.... with an hour or so of loop gain tweaking it pretty much drives the course as programmed.
The one thing I have left to do is to program acceleration and deceleration into the Doping class car... so I take a bunch of data on how hard it accelerates/decelerates. What speeds it can corner without slipping, how much brake is necessary to decelerate well etc.... given all this data I sit in the parking lot across the street and try to implement this....its all pretty much done when a security guard asks me to leave.
I tell hip its an empty parking lot nd I only need about 10 more minutes. ?he says as he makes his rounds he will be back in 5 min... I have till then...
So I hurry things up and go out to test the car.. in a hurry I put the new code in the car and rushing to get the car tested I drop the RC transmitter on the tailgate of my truck.... while holding the car. The car goes to 100% throttle... (Over 100mph with this car) so I'm holding a car screaming like a banshee with significant gyro forces trying to tear it out of my hands. I reach down and slam the throttle stick to full brake.
This is more than the speed control can take an the car, batteries and speed control errupt in flames.
Its now 10:15 pm the day before the race and the car just caught fire.
Back to the hotel room to strip the speed control off of last years car In the spares there is a similar speed control ..I dissemble the car and install that.
His speed control is wired to a single battery connector and my ole one was wired for dual.. as my list of things to do included moving the battery for better balance I do that at this time too...
The battery voltage changed from 6S to 4S and the speed control changed from Mamba Monster to Mamba Monster II. I download the castle creations programming software to the lap top and try to reprogram the speed control into linear mode like the last one.. alas it does not support this mode.
So all the acceleration deceleration mapping data I took is now invalid.
Its midnight I'm out in an empty theater parking lot redoing the acceleration/deceleration curves.
I get this done and test the code.. a few tweaks later the car is running pretty good. Its insanely fast and I'm really happy with the results. I decide to do one last run takes off really fast, makes the first corner ... turns left accelerates between corners at the second corner it does not slow down.... it doesn't' turn it just goes straight and disappears from view as it leaves the area of lit parking lot at full speed....
Its now 1:30 am and I'm wandering a huge dark parking lot searching for my car... after about 15 minutes I find it, it hit the curb at speed and the left front shock mount snapped off the bolt holding it on and is dangling loose. The error is mine.. I let the battery that runs the GPS and CPU get too low and the GPS stopped reporting. I go back to the hotel and dig through my spares.. I brought spares of all the suspension components... except the bolts that hold them together..... I head out to the all night walmart hoping they have bolts.... their bolt selection is really weak and and back at the hotel at 3am I end up stealing screws from other parts of the car to fix the broken parts. Its 3:30 am I'm getting up at 6:30 to go to the event... I think the car is ready... I still have to strip out the code that listens to the RC receiver... I'll do that in the morning. Much better than last year I get 3 hours of sleep. (As compared to 0)......
To be continued I'll cover the day of the race in my next post.
The one thing I have left to do is to program acceleration and deceleration into the Doping class car... so I take a bunch of data on how hard it accelerates/decelerates. What speeds it can corner without slipping, how much brake is necessary to decelerate well etc.... given all this data I sit in the parking lot across the street and try to implement this....its all pretty much done when a security guard asks me to leave.
I tell hip its an empty parking lot nd I only need about 10 more minutes. ?he says as he makes his rounds he will be back in 5 min... I have till then...
So I hurry things up and go out to test the car.. in a hurry I put the new code in the car and rushing to get the car tested I drop the RC transmitter on the tailgate of my truck.... while holding the car. The car goes to 100% throttle... (Over 100mph with this car) so I'm holding a car screaming like a banshee with significant gyro forces trying to tear it out of my hands. I reach down and slam the throttle stick to full brake.
This is more than the speed control can take an the car, batteries and speed control errupt in flames.
Its now 10:15 pm the day before the race and the car just caught fire.
Back to the hotel room to strip the speed control off of last years car In the spares there is a similar speed control ..I dissemble the car and install that.
His speed control is wired to a single battery connector and my ole one was wired for dual.. as my list of things to do included moving the battery for better balance I do that at this time too...
The battery voltage changed from 6S to 4S and the speed control changed from Mamba Monster to Mamba Monster II. I download the castle creations programming software to the lap top and try to reprogram the speed control into linear mode like the last one.. alas it does not support this mode.
So all the acceleration deceleration mapping data I took is now invalid.
Its midnight I'm out in an empty theater parking lot redoing the acceleration/deceleration curves.
I get this done and test the code.. a few tweaks later the car is running pretty good. Its insanely fast and I'm really happy with the results. I decide to do one last run takes off really fast, makes the first corner ... turns left accelerates between corners at the second corner it does not slow down.... it doesn't' turn it just goes straight and disappears from view as it leaves the area of lit parking lot at full speed....
Its now 1:30 am and I'm wandering a huge dark parking lot searching for my car... after about 15 minutes I find it, it hit the curb at speed and the left front shock mount snapped off the bolt holding it on and is dangling loose. The error is mine.. I let the battery that runs the GPS and CPU get too low and the GPS stopped reporting. I go back to the hotel and dig through my spares.. I brought spares of all the suspension components... except the bolts that hold them together..... I head out to the all night walmart hoping they have bolts.... their bolt selection is really weak and and back at the hotel at 3am I end up stealing screws from other parts of the car to fix the broken parts. Its 3:30 am I'm getting up at 6:30 to go to the event... I think the car is ready... I still have to strip out the code that listens to the RC receiver... I'll do that in the morning. Much better than last year I get 3 hours of sleep. (As compared to 0)......
To be continued I'll cover the day of the race in my next post.
Monday, May 06, 2013
Cute coding trick...
In building cars and planes and rockets, I collect a lot of data log files for all of the projects....
I backup all the source files using source control on every build..., but I presently had no way to connect a data file to the exact source that was used to make it... I'm sure this has been done a thousand times... before but I reinvented the wheel last night...
I wrote a simple header file...
call it FileList.h
//A Class to make a static linked list of ALL files included in the project....
class FileList
{
private:
static FileList * pHead;
FileList * m_pNext;
const char * m_pName;
public:
//The constructor will add the name to the global static linked list...
FileList(const char * name)
{m_pNext=pHead;
pHead=this,
m_pName=name;
};
static void ShowList();
};
//A Static opbject that does the real work...
//__BASE_FILE__ is the name of the file being compiled rather thatn this header file
static FileList FL(__DATE__ "," __TIME__ "," __BASE_FILE__);
I backup all the source files using source control on every build..., but I presently had no way to connect a data file to the exact source that was used to make it... I'm sure this has been done a thousand times... before but I reinvented the wheel last night...
I wrote a simple header file...
call it FileList.h
//A Class to make a static linked list of ALL files included in the project....
class FileList
{
private:
static FileList * pHead;
FileList * m_pNext;
const char * m_pName;
public:
//The constructor will add the name to the global static linked list...
FileList(const char * name)
{m_pNext=pHead;
pHead=this,
m_pName=name;
};
static void ShowList();
};
//A Static opbject that does the real work...
//__BASE_FILE__ is the name of the file being compiled rather thatn this header file
static FileList FL(__DATE__ "," __TIME__ "," __BASE_FILE__);
If you #include "filelist.h"
in every single C++ file in your project... this will make a linked list of
all the file names, dates and times..... a list I can prepend to the data log on startup...
so now the whole code set and its date time are included in the data log on every start...
Maybe this is common knowledge, but the __BASE__FILE__ macro was a new one to me...
Monday, April 29, 2013
Data logging...
Logging data from "Things", I've run computer operated planes, rockets, cars, helicopters. All of these share a similar problem. How do you log data? I like to log ALL raw sensor readings unmodified. This ends up in a steam of multi rate data, ie I see raw IMU readings at 200Hz, Compass readings at 20Hz GPS at 20Hz, and various things like mode changes and responses at varying rates from 200Hz to to once at power up.
Some things I've learned:
Where START is a specific byte and ID is a numeric value representing the kind of data...
If the value START appears in the data is is escaped...The packet ends with the next start....
I then have a background process that takes this circular buffer and writes it to flash sectors where there is a full sector worth of data.
Now I might record something like my GPS reading...
typedef struct
{
double lattitude; //Radians
double longitude;//Radians
double ht; //m
unsigned long msec;
unsigned short week;
double ECEF_X;//m
double ECEF_Y;//m
double ECEF_Z;//m
float hspeed;//m/sec
float head; //radians
float vv; //m/sec
unsigned char flag1;
unsigned char flag2;
unsigned char nsat;
WORD seq;
}CombinedTrimGps;
So that would be stored as
START GPS_ID the contents of the structure.....
Now if in the past if I changed the record so I added say a list of SNR reading to the record I'd have to redo the code that decodes the GPS data from the data logs....
In an ideal world C++ would have introspection and it would be able to put the format of the structure into the data log the first time its used..... then when the data reader sees the data it immediately knows what format the data is in....
Since C++ does not have introspection... I wrote some code that comes very close...
void LogRecord(CombinedTrimGps & item)
{
LogStart(LOG_TYPE_BD960 ,"GPS");
LogElement(lattitude,"lattitude " );
LogElement(longitude,"longitude " );
LogElement(ht ,"ht " );
LogElement(msec ,"msec " );
LogElement(week ,"week " );
LogElement(ECEF_X ,"ECEF_X " );
LogElement(ECEF_Y ,"ECEF_Y " );
LogElement(ECEF_Z ,"ECEF_Z " );
LogElement(hspeed ,"hspeed " );
LogElement(head ,"head " );
LogElement(vv ,"vv " );
LogElement(flag1 ,"flag1 " );
LogElement(flag2 ,"flag2 " );
LogElement(nsat ,"nsat " );
LogEnd();
}
Using Macros this expands into...
void LogRecord(CombinedTrimGps & item)
{
static bool bIntroed;
BYTE tid=LOG_TYPE_BD960;
if(!bIntroed)
{
bIntroed=true;
StartItemIntro(tid,sizeof(item),"GPS");
ShowElement(tid,&item,&item.lattitude,item.lattitude,"lattitude " ) ;
.
.//All the elements
.
}
LogRawRecord(tid,(const unsigned char *)&item,sizeof(item));
}
I have a different ShowElement overloaded function for each type I might want to log....
Thus the first time this data type is logged it puts the format into the data log record....
This is something I've wanted since I first did the rocket stuff 6 years ago.
I've now got a self describing log format that is complete and efficient....
I can also use the exact same format for Telemetry....
On my rockets I had an optional parameter that went to the logging function that selected if the data went to the LOG, to telemetry or to both locations....
Last night I got the whole chain to work and the picture below is derived from this chain of events...
IE I drove the AVC car around my driveway -> recorded log -> process log generating records -> put in excel -> Then made a graph of locations....
Some things I've learned:
- I don't trust a file system. If your thing crashes the data you are really interested in is the last few bits of data before impact. I've lost too much data to trust a file system to be robust when the recording media is de-powered and spattered all over the landscape
- You can't squeeze everything you want to record in a telemetry stream.
- I keep reinventing this same wheel.
- In the course of a project the format of what you are recording evolves. Keeping the code for the data display and the data recorder in sync is difficult and often I end up unable to read earlier logs.
My current approach is as follows:
Record everything to flash directly, it makes no difference if its a SD card or dedicated flash chip I use the same format. I fully erase the flash (all the sectors on SD) then on power up I scan the flash page or sector by sector looking for the first blank page/sector. I then start recording new data in that sector....
My data recording has a big circular buffer that records the data in stream format....
All the recorded data is in blocks with the format:
START ID ... (Next START)
Where START is a specific byte and ID is a numeric value representing the kind of data...
If the value START appears in the data is is escaped...The packet ends with the next start....
I then have a background process that takes this circular buffer and writes it to flash sectors where there is a full sector worth of data.
Now I might record something like my GPS reading...
typedef struct
{
double lattitude; //Radians
double longitude;//Radians
double ht; //m
unsigned long msec;
unsigned short week;
double ECEF_X;//m
double ECEF_Y;//m
double ECEF_Z;//m
float hspeed;//m/sec
float head; //radians
float vv; //m/sec
unsigned char flag1;
unsigned char flag2;
unsigned char nsat;
WORD seq;
}CombinedTrimGps;
So that would be stored as
START GPS_ID the contents of the structure.....
Now if in the past if I changed the record so I added say a list of SNR reading to the record I'd have to redo the code that decodes the GPS data from the data logs....
In an ideal world C++ would have introspection and it would be able to put the format of the structure into the data log the first time its used..... then when the data reader sees the data it immediately knows what format the data is in....
Since C++ does not have introspection... I wrote some code that comes very close...
void LogRecord(CombinedTrimGps & item)
{
LogStart(LOG_TYPE_BD960 ,"GPS");
LogElement(lattitude,"lattitude " );
LogElement(longitude,"longitude " );
LogElement(ht ,"ht " );
LogElement(msec ,"msec " );
LogElement(week ,"week " );
LogElement(ECEF_X ,"ECEF_X " );
LogElement(ECEF_Y ,"ECEF_Y " );
LogElement(ECEF_Z ,"ECEF_Z " );
LogElement(hspeed ,"hspeed " );
LogElement(head ,"head " );
LogElement(vv ,"vv " );
LogElement(flag1 ,"flag1 " );
LogElement(flag2 ,"flag2 " );
LogElement(nsat ,"nsat " );
LogEnd();
}
Using Macros this expands into...
void LogRecord(CombinedTrimGps & item)
{
static bool bIntroed;
BYTE tid=LOG_TYPE_BD960;
if(!bIntroed)
{
bIntroed=true;
StartItemIntro(tid,sizeof(item),"GPS");
ShowElement(tid,&item,&item.lattitude,item.lattitude,"lattitude " ) ;
.
.//All the elements
.
}
LogRawRecord(tid,(const unsigned char *)&item,sizeof(item));
}
I have a different ShowElement overloaded function for each type I might want to log....
Thus the first time this data type is logged it puts the format into the data log record....
This is something I've wanted since I first did the rocket stuff 6 years ago.
I've now got a self describing log format that is complete and efficient....
I can also use the exact same format for Telemetry....
On my rockets I had an optional parameter that went to the logging function that selected if the data went to the LOG, to telemetry or to both locations....
Last night I got the whole chain to work and the picture below is derived from this chain of events...
IE I drove the AVC car around my driveway -> recorded log -> process log generating records -> put in excel -> Then made a graph of locations....
Now I have to get the data replay tool I wrote a few years ago to read this format and my data logging tools will be complete!
Friday, April 19, 2013
Finding a Firmware bug...
I spent the last few days hunting down a bug.
It was interesting enough that I thought I'd write a short blog post....
First some background...
In all of the NetBurner products we try very hard to make sure address space 0 is unmapped,
ie any attempt to reference a NULL pointer should cause an error.
The particular processor I was bug hunting on boots at address zero so in most normal scenarios address 0 is mapped to some kind of memory.
In our case this is not so...after the boot monitor boots, it turns off the hardware memory mapping to address 0...and relocates all valid memory up higher in the memory map...
(We are not using an MMU)
The bug we were hunting was an interrupt latency bug, 99.99% of the time the part has an interrupt latency of 0.9 usec or so... every once in a while it would have a longer latency, randomly varying from 1 to 520usec. 520usec is WAAAAAY too long.....
So to instrument this I setup one of the on chip timers to make a pulse and reset the counter at a fixed rate...
I hooked this timer output up to the non maskable interrupt and in the interrupt the first thing I did was read the timer counter value... this is an all on chip direct measurement of the interrupt latency...
And soon found that it varied randomly from 0.9 to 520usec.
Step 1 complete we can replicate the problem.....
After this I tried a whole bunch of things to hunt this down, looked at code, moved stacks and vector tables to/from different classes of memory... nothing changes the result....
If I make the whole code set small enough so that it all fits in the instruction cache it goes away....
As a random idea I changed the bus timeout monitor from very long to something shorter...
and the latency improved... this was the clue I needed to hunt down the bug....
Here is the bug:
We use a lot of C code in the system of the form:
It was interesting enough that I thought I'd write a short blog post....
First some background...
In all of the NetBurner products we try very hard to make sure address space 0 is unmapped,
ie any attempt to reference a NULL pointer should cause an error.
The particular processor I was bug hunting on boots at address zero so in most normal scenarios address 0 is mapped to some kind of memory.
In our case this is not so...after the boot monitor boots, it turns off the hardware memory mapping to address 0...and relocates all valid memory up higher in the memory map...
(We are not using an MMU)
The bug we were hunting was an interrupt latency bug, 99.99% of the time the part has an interrupt latency of 0.9 usec or so... every once in a while it would have a longer latency, randomly varying from 1 to 520usec. 520usec is WAAAAAY too long.....
So to instrument this I setup one of the on chip timers to make a pulse and reset the counter at a fixed rate...
I hooked this timer output up to the non maskable interrupt and in the interrupt the first thing I did was read the timer counter value... this is an all on chip direct measurement of the interrupt latency...
And soon found that it varied randomly from 0.9 to 520usec.
Step 1 complete we can replicate the problem.....
After this I tried a whole bunch of things to hunt this down, looked at code, moved stacks and vector tables to/from different classes of memory... nothing changes the result....
If I make the whole code set small enough so that it all fits in the instruction cache it goes away....
As a random idea I changed the bus timeout monitor from very long to something shorter...
and the latency improved... this was the clue I needed to hunt down the bug....
Here is the bug:
We use a lot of C code in the system of the form:
if ( pDHCPTick )
{
pDHCPTick();
}
This code checks a function pointer to see if its null and calls the function if it isn't...
This chunk of code is inside one of the system tasks and is used to service the DHCP client if its active...
This compiles to assembly code something like:
moveal pDHCPTick,%a0
testl %a0
beqw skip
jsr @%a0
skip:
So if pDHCPTick is null the jsr @%a0 will never be executed....
But the processor I'm having the issue with has aggressive pipelining....
So it tries to load the @%a0 even if it does not call it.
This causes an instruction fetch of address 0....
If address 0 is not in the instruction cache then this forces a cache miss and a cache line read from physical address 0.....
Physical address 0 is un-mapped so the bus timeout monitor goes off terminating the transaction 520usec later...
If the interrupt has the misfortune of trying to go off while we are hung in the bus time out we get long latency....
So the problem was solved by setting up the system so there is a valid bus ack at address 0....
Makes it so we don't catch NULL pointer accesses... but fixes the interrupt latency...
We are working on a better solution.. but I thought the problem crossed enough different hardware/software domains where it was interesting.
Wednesday, April 03, 2013
Teaching a computer to fly...
For this years spark fun AVC I'm planning to bring both a Car and a Plane. The Car is primarily last years car upgraded with a lower center of gravity and a much much better GPS. The plane is a different story...
As part of the Lunar Lander Challenge I built an autonomous helicopter to test out my code. Right now I have a vectored thrust rocket I'm flying as well, all in all these two efforts have the actuators and response of the vehicle fairly well coupled and tight. a rocket or helicopter is a brute force thing that beats the environment into submission. An airplane is a gentle elegant creature. I 've been a pilot since I was a teenager, and when I look at most of the open or openly discussed UAV autopilots out in the world it strikes me that a lot of the people working on them don't understand this distinction.
I have a copy of x-plane 10 with I'm using as a simulator for the autopilot development.
I've set this up as a hardware in the loop simulator and I've been slowly teaching the NetBurner to fly.
Much like my Dad taught me to fly a long time ago.
- Manage your airspeed,
- Coordinate your turns,
- Hold a heading,
- Hold an altitude,
- Make smooth level turns...
- Turn to a heading...
Now learn to do all of the above while climbing, descending, accelerating, decelerating, you have a precision tool with the grace of a light saber, feel the grace. If the engine dies it should still be in control when glides gently to earth.....
An airplane can be flown with a single axis rate gyro, anyone that has passed a partial panel IFR checkride understands this. I have all the rate gyro integrating quanterion attitude estimaters that everyone else has.... alas most of this is not needed you can do everything you need to do to control the airplane with airspeed , rate of turn, magnetic compass and altitude.
The interactions can be subtle... does the throttle control speed? does it control altitude? what variable does it control? How about energy..... I found this paper to be especially good on this point.
The basic bottom level control laws are simple....
Inputs:
Desired Airspeed
Desired Heading
Desired Altitude (or rate of climb)
Sensors:
Pitot AirSpeed (IAS)
Magnetic heading
Rate of turn.
Altitude (either baro or GPS driven)
The controls are simple...
The elevator is controlled with Airspeed and TargetAirspeed
The rudder does nothing more than keep the "Ball" centered in flight or the aircraft on the runway on takeoff.
The Aileron is controlled by rate of turn, magnetic heading and target heading
The throttle is controlled with the current energy state ie airspeed ^2 and altitude vs their target values.
(I've also found it useful to have a bit of feed forward from turn rate to throttle to add a little power in the turns)
In any case I'm having fun writing this....
Tuesday, October 30, 2012
The Business plan info...
I'm on the space show tonight and I have been meaning to make the documents from the NewSpace business plan competition public... so here they are.
The goal was to assume the technology worked and to emphasize the business aspects.
So those that are looking for lots of technical details will be disappointed...
The Executive summary.
The Full Plan. (Written in a hurry while I had a 103 deg fever.)
The Slide deck... zipped up...
The goal was to assume the technology worked and to emphasize the business aspects.
So those that are looking for lots of technical details will be disappointed...
The Executive summary.
The Full Plan. (Written in a hurry while I had a 103 deg fever.)
The Slide deck... zipped up...
Monday, June 18, 2012
A Car, Beta dog food and coding in the block house.
This last weekend I competed in the Sparkfun Autonomous vehicle competition. I did this with a 1/5 scale RC car, a brand new NetBurner platform (NANO54415) and a lot of last minute coding. (My son and I call this last minute coding coding in the blockhouse.) As I was leaving the house to catch the plane to Denver my son asked my if I was ready and how was it going. My one sentence answer was I'll be coding in the block house. This refers to all of our rocket testing, one really wants to show up on site with everything done and ready to go. Reality does not always work that way and final software is almost always the last thing that gets finished. This case was very similar....
On to my story...
A month or so ago I had all of the Major car hardware working and shot this video
This demonstrated that all of the basic hardware was working....
I then started working on the video vision system to find the bonus Arch.
I was capturing video, converting to red only and then edge finding... all of this was working
when spark fun published the course preview saying the arch would be fixed on one position so one could find it with precision navigation, no need to find it... Argh!!!! I'd spent weeks working on the vision and building a duplicate arch etc....
So at this point I sort of stopped working on the Car and tried to get my airborne entry flying.
This was my first strategic blunder, trying to do too much.
I was trying to do too much and on June 4th I gave up on the Plane and went back to the car.
In the process I wrote a simulator....
I also started driving around the a nearby building that was in a similar configuration to the SparkFun building. In driving around this building I learned several things:
1)My test case in the parking lot had really clean walls, the test building walls were less clean, and had a section of windows where the Laser Range finder sometimes went through and sometimes did not.
2)I really needed to add a wheel turn counter so I could control precisely where to wall follow and where to just do dead reckoning.
3)As I suspected the low cost L1 GPS was not consistently precise enough to drive around the building reliably.
At this point I made the second biggest strategic blunder of the contest.
I got the impression that spark fun AVC was a pretty low budget hacker event for most participants. In my leftovers from the LLC project I have a pair of Trimble BD950 L1,L2 survey grade RTK GPS receivers. I also have a pair of survey grade choke ring L1,L2 Omnistar capable antennas. I thought about bolting this GPS to the car and using that for navigation. I personally thought that using a $5K GPS receiver setup while legal would be un-sportsman like. (Note that the car that won used the same chassis I did and a really nice expensive GPS receiver. So lesson learned don't handicap yourself, if you have an advantage use it.....
So I went back to the shop and machined a couple of mounts to mount an LED and photo transistor looking through the main drive gear on the Car. I then hooked this up to an interrupt pin on the NetBurner and started counting pulses.... only this was not really reliable... and out comes the scope...
The photo transistor was not reliably pulling the signal to logic low. It was only going to about 1.2V and if I changed the resistive pull up it go noisy... So I move the input from the interrupt to an A/D input. Only to discover that on our beta release for the Netburner Nano54415 the only A/D example was polled not interrupt driven. So since it was really my day job to write all the divers for the Nano time to write an interrupt driven a/D system for the nano... the first attempt ended up with a single channel A/D conversion interrupt rate at 1.2Mhz seemed like over kill for a pulse whose maximum rate was going to be 500hz or less.... so I turned the A/d clock way down to its minimum value and added in the other 8 A/D channels (only 6 are pined out the chip has 8) and got the rate down to 20Khz or so. The Odometer now was 100% perfectly reliable. (This is the eat your own dog food part.. Ie at Netburner we don't just make modules, we actually try to use them on a regular basis to do things so we understand the customers point of view)
In fact I did a trip around my test building and the dead reckoning was pretty good..
I changed my data logging format so I don't have a copy of the DR trace from my first DR run around the local building, I'm posting one from Sparkfun instead)
Here is a DR plot of two laps around the spark fun building Thursday night.... I drove from where my car was parked in the street out front and went around the building twice...
The DR was almost good enough to to the task all by itself.... I was off about 15ft per lap.
I then started setting up Garbage cans in my drive way and using the LASER range finder to dodge them while staying on course.... I learned that having the laser scanner rotate 90 degrees
to look ahead while also measuring the side range for navigation was problematic. It was fast enough, but the timing of the servo drive and the main command loop were asynchronous enough that I was never really sure where the scanner was pointed and could not properly associate range results with what heading they were on... As I planned on having spare of most everything I then took my spare range finder and mounted it 90 degrees to primary range finder... so now I had two range finders
This was the final physical I/O configuration...
While using the laser range finder to map barrels in my drive way I accidentally launch the car down the drive way.... still attached to my laptop thus ripping the ethernet jack out of the back.
Ebay to the rescue I got a super rugged lap top from Ebay three days later I had the joy of moving all my tools and code from old laptop to new laptop with out the benefit of ethernet.
I am now very much running out of time. On Tuesday the data from the side laser range finder is really flaky So Wednesday Morning I order another range finder so I have a spare and have it shipped to my hotel room in Boulder.
Wednesday night I go back out to my nearby test building and drive around a few times collecting data. I then spend Wednesday evening fine tuning my data reduction method. For my rocket project I'd written a custom data display application and that was a huge effort. As this seemed to be a simpler project I used a different approach...
The logging was almost identical, IE I logged everything. I stated out by logging to the SD card, but managing file names and when to flush the buffers etc... was a hassle, as I have 64M of available RAM I just logged to RAM and then at the end of the run FTP'ed that data set to the lap top. Quicker and easier than the SD card in this case. For a vehicle that might physically crash such as an airplane or helicopter, or rocket my strategy is to write to the SD card physical sectors directly so the data is always fresh, in this case with the car it did not really apply so I just left it in RAM. (More on the cost of this decision later)
So to reduce the data I wrote a program that would convert the binary log file in to a CSV file and look at it with excel I could do linear plots and scatter plots etc... worked, well but it was slow as to add plots or graphs I had to manually go though the excel interface and add charts...
I eventually figured out how to highlight specific ranges of the data on the scatter plot giving me the ability to tie a specific piece of data to a specific physical location. (See the DR plot above the Pink is a highlighted section of data. Properly viewing multiple rate data is (for me) an unsolved problem.
My Rocket display code worked well, but it was purely linear display, no way to tie the data to a physical location or map. I should probably write something to permanently put in my toolbox for this as its a thorn that keeps poppoing up over and over... either that or learn to wrote VBA code for excel.(yuck).
So Wednesday night I pack up everything, all the spares the car 4 sets of laptop batteries, etc...
I only had one set of Lipos for the car as I was concerned about bringing big Lipos on the plane so the previous week I'd sent 3 sets of Lipos for the car to my hotel in boulder UPS ground.
Soldering tools, a spare everything....
(About 6 pm Wednesday I realize that while I have a spare IMU its got the wrong code in it, so I make a trip to my office where the code load for the IMU Was and reprogram the spare)
I get to bed by midnight or so... and leave the next morning at 5:30 am to catch the plane.
My flight leaves at 7:45 but I'm checking and carrying a lot of strange bits and pieces so I want to allow lots of time. The check in goes fine and the security line is huge, I have my wife waiting in the waiting lot to take the Lipos from me if they don't clear security.
Good thing I had her wait as I go through security it dawns on me I've only gotten three of the four boxes and bags I needed, and I left one in the trunk of the car.... I call her pick that up and wait in line again. Security lets me through.... does not even glance at the 4 laptop batteries or the 3 Large Lipos in my bag.
I arrive in Denver and go to get my rental car.... I have a reservation alas the previous week there was hail in Denver damaging more than half of the rental car fleet. it takes four hours to get my car. There were people in the back of the rental car line that had reservations for a car and weren't going to get a car. There just weren't enough available...
So I drive to boulder, eat some lunch and check into my hotel room. I unpack everything and assemble the car. I drive it around the hotel parking lot to make sure that all of the sensors are working... and head for Spark fun... its clear that they are setting up for the AVC, but there are too many cars in the parking lot to take any useful data to I walk around mentally looking at things and go back to my room. I return to Spark fun around 9 pm that night after all the Spark fun people have left and I'm one of three cars testing at Spark fun.
I drive the car around the building a few times recording all the laser ranges and GPS.
the side facing Laser range finder is still flaky even after swapping out the unit.
So I point the good laser range finder to the right and take two more runs around the building recording that.
I go back to the car make sure I've captured good data and go back to the room.
I get to bed around midnight and get up at 6:15 Friday morning.
I reduce all the data I took into a set of GPS coordinates and a set or range heading and odometer readings. In the last week the AVC project has destroyed my training schedule for the Half Ironman I want to do on my 50th birthday in September , so I use an hour or so to clear my head and get some exercise. At around 9 am I go find the Boulder recreation department public pool and swim about 1.5 miles. Side note swimming is usually a bit hypoxic to begin with, swimming at 5000+ ft of elevation and at fast training pace is hard...)
I go find Lunch and then head back to the room. There is a big empty parking lot that is blocked from traffic near the hotel. I'm not sure what it was for, but it had barriers at the entrance and a bunch of light poles with big concrete bases that were goo barrel dodging simulations.
For practice I drive around four poles and reduce that data to a program that would navigate around the barrels using the barrels and laser range to find the corners... This is working well and then I start testing the barrel dodging code and it all stops working.... it seems that I let the primary battery get too low and the Laser range finders all lost their calibration ARGHHHH! Its now 2 pm and Spark fun has a tour at 3PM, I really want to take the spark fun tour so I drive out to Sparkfun and take the tour. Back at the hotel at 4:30 pm and work on re calibrating the Laser range finders.... I decide to switch the laser range finder to raw count uncalibrated mode and just use raw counts an do the cal myself...
Here I realize what is probably Strategic blunder #0 assuming that other embedded programmers did their job. The Data sheet for the Laser range finder says accurate to +/- 1 Yd no mention of resolution, yet the data print out in Ft mode was DIST:12.3F IE it had lots of resolution, or so I thought... it seems the absolute raw resolution is about 0.5 yd's and the data after the decimal point was 100% meaning less. I had much less resolution that I though I had and finding the arch based on this distance was going to be dicey....
I'm now really worried that the Laser range finding navigation scheme is not going to work at all.
So I do what I should have done weeks ago in San Diego, I write the GPS following code to have a back up plan. I go out to my test parking lot and take GPS coordinates and test this code I successfully drive around and around the parking lot using GPS. It works real well and is more accurate in the open parking lot than it was near my test building.
So at this point its about 9pm and I have a Laser nav program and a Gps nav program set up for Spark fun... both untested on site.... I stop at McDonalds get a burger and a very large Ice tea caffeine supply and proceed to Spark fun.
There are about 5 groups driving around the building with their car.
Between 10PM and midnight I see one group drive their car into the pond twice.....
The first half of the course works well almost immediately....
Finding the first corner is a bit dicey and the spark fun people have put up 2" diameter posts with a 8" diameter base all around the building on the edge of the sidewalk. The laser scanner keeps catching glimpses of these and thinking its too close to the building and as a result it heads for the pond..
My RC kill switch is fast and I save it several times... So I tune the algorithm to take 16 point averages and only use the ranges from the highest 8 or so readings... I also start setting up exclusion zones where the system ignores the laser and just drives on heading... for some fixed distance.
My first around the building state machine had 8 states...
Numbers are odometer counts, reset means set odo back to zero...
/*
DR Version...
//Heading 85 deg Wall follow at 25 ft at 750 arm
//Hold heading till distance jumps by 20 ft
//Then coast 300
//Then turn to 167 Reset Odo
//Then coast 100
//Then follow wall at 28 ft till odo 2000.
//Hold Heading wait till distance jumps to 35 reset odo
//Coast 600
//Then -105 Deg reset odo
//Coast 200
//Wall follow at 42 Arm at odo 1100
//Hold Heading wait for Jump to 52
//Turn to -18 deg Reset Odo Coast 100
//Then wall follow at 20 ft. Arm at 2200
//When distance goes to 30 fto turn to 85 deg
//Coast 100
//Wall follow at 25 ft till odo 1200
//Done.
*/
I have two problems, First the parking lot is not flat, in some places its enough not flat that the car tips enough that the side laser scanner (still being flaky) points at the ground not the building....
So with a bit a cardboard and tape I re point the laser scanner up about 15 degrees.... things get much more consistent.
I keep having a problem with the third turn, no tuning I do seems to make it work. Its now about 2am and I'm at less than peak performance....
So I switch to my GPS code and tune that a little, about 8 tries later the GPS version drives around the building, but in the next three attempts it only works 2/3 of the time. The variance in the low cost GPS is just too much... I save this code as a back up and go back to laser scanner mode...
I immediately see what my issue is I forgot a break in a case statement, so no matter how much I told it to extend out the third turn it just went to the next state turn ... I fix this and a similar copy and paste error two steps later. Next attempt it extends the third turn out way too far and changing this back and now we make it almost all the way around the building.. Fix up the forth turn and we finally make it all the way around the building autonomously with out error. We do this twice more...
I've been runnign after the car all night to see what it is doing. so its speed is set to as fast as I can run... as the night wears on the speed gets set slower so I can keep up.
So I turn the speed back up start the car, watch to make sure it gets past the pond and then I go back the other way to see it come out... no car....
I find it in a bush very far from where it should be....
I try again same result slightly different bush...
I turn the speed back down to where it worked...
Its in a different bush...
So I now run after it.... it works perfectly....
Again I run after it works perfectly.... my knees are starting to hurt from doing wind sprints all night...
I try to launch it and not run after it... strange bush... doesn't work.
Then it dawns on me... when it looses sight of the RC transmitter the RC receiver goes into fail safe mode and stops the car... then when I come looking for it it sees the RC transmitter before I see it and the RC Tx is still in autonomous mode so the car sees this as a manual -> autonomous toggle and starts thinking its at the beginning of the course!!!!! ARghhhh! So I turn the speed up a little and walk ahead of the car as far as I can still retain manual control.... I then start the car and run as fast as I can and it goes around the building... whooo its all working.... then the laser scanner stops working at all!!! Its starting to get light and the first person to show up around 5am with their car says to me early morning or late night.... I make some kind on unintelligible response and one look at me makes him worry that the zombie apocalypse has come to Boulder....
The flaky Laser sensor has felt like bad wiring from the very beginning, if disassemble the housing and hook it directly up to the laptop serial port and power it with an external battery it works every time. I've pulled tugged, and re soldered all the wires from the LASER range finder back to the dev board about 2000 times and can never find a bad connection.....
It starts working again.... but in the process I break the glue joint on the mount....
So I need some glue to re glue the mount and ask the spark fun people (now here setting up) where to find a hardware store and breakfast near by... Or where can I find power to use the Hot Melt Glue gun in my took kit ... They turn on power to the pits area and I hot melt glue the laser scanner mount back together... laser scanners are not both working...
At this point the 4th laptop battery is dead and I go to my car to get the charger only ot realize its in my room at the hotel...
Into the car and drive back to the hotel stopping at the same McDonald's to get breakfast.
Get the charger, change my shirt and try to wipe the grime off my face and hands...
Back to spark fun... its now 8:30 and the contest starts at 9.
I charge the laptop and put fresh Lipos on the car. Sitting on the bench test everything the side laser scanner is not working again... I see a loose wire on the board... The carrier board I'm using only has level translators for one RS-232 serial port. The IMU and GPS are TTL levels, the laser scanners are RS-232 so to get around this I jumper the CTS in line from the one port from the CTS pin to the RX pin on a different serial input (The board has 8 serial ports pinned out) My flaky wiring for the side laser scanner was a bad crimp on the jumper from the CTS to the RX pin... not on the external wiring at all. I fix this and never again have a laser scanner problem for the rest of the day.
I turn the speed down to the speed I can run and the first heat goes reasonably well... we make it 3/4ths of the way around the building the light has changed and now the laser scanner can see through the window on the last turn, it sees this jump in range and thinks the corner of the building has passed and turns into the building....
So I extend the lock out range for corner detection and on the second heat still in slow as I can run mode we make it all the way around in 1:06 We miss the arch so no arch bonus...
(I think this run was good for 4th or 5th place)
This photo was taken by my Friend Ben Brockert (@Wikkit) between the 2nd and third heat.
The last heat... I turn the barrel dodging code back on as last time we just plowed through the barrels... I turn the speed way up and change the code to ignore the RC receiver after going into autonomous mode..... The car hauls down the straight, properly dodges a couple of barrels go straight through the arch and around the corner I run to the other side and it never comes out....
So I walk back and find it in my favorite bush at this point the competition is over and I have a choice... I can carry the 20lb car back to my laptop and download the data log to see what happened... or I can hit reset and manually drive the car back and loose the data log never knowing for sure what happened on the last lap.... I'm tired I press reset....
On to my story...
A month or so ago I had all of the Major car hardware working and shot this video
This demonstrated that all of the basic hardware was working....
I then started working on the video vision system to find the bonus Arch.
I was capturing video, converting to red only and then edge finding... all of this was working
when spark fun published the course preview saying the arch would be fixed on one position so one could find it with precision navigation, no need to find it... Argh!!!! I'd spent weeks working on the vision and building a duplicate arch etc....
So at this point I sort of stopped working on the Car and tried to get my airborne entry flying.
This was my first strategic blunder, trying to do too much.
I was trying to do too much and on June 4th I gave up on the Plane and went back to the car.
In the process I wrote a simulator....
I also started driving around the a nearby building that was in a similar configuration to the SparkFun building. In driving around this building I learned several things:
1)My test case in the parking lot had really clean walls, the test building walls were less clean, and had a section of windows where the Laser Range finder sometimes went through and sometimes did not.
2)I really needed to add a wheel turn counter so I could control precisely where to wall follow and where to just do dead reckoning.
3)As I suspected the low cost L1 GPS was not consistently precise enough to drive around the building reliably.
At this point I made the second biggest strategic blunder of the contest.
I got the impression that spark fun AVC was a pretty low budget hacker event for most participants. In my leftovers from the LLC project I have a pair of Trimble BD950 L1,L2 survey grade RTK GPS receivers. I also have a pair of survey grade choke ring L1,L2 Omnistar capable antennas. I thought about bolting this GPS to the car and using that for navigation. I personally thought that using a $5K GPS receiver setup while legal would be un-sportsman like. (Note that the car that won used the same chassis I did and a really nice expensive GPS receiver. So lesson learned don't handicap yourself, if you have an advantage use it.....
So I went back to the shop and machined a couple of mounts to mount an LED and photo transistor looking through the main drive gear on the Car. I then hooked this up to an interrupt pin on the NetBurner and started counting pulses.... only this was not really reliable... and out comes the scope...
The photo transistor was not reliably pulling the signal to logic low. It was only going to about 1.2V and if I changed the resistive pull up it go noisy... So I move the input from the interrupt to an A/D input. Only to discover that on our beta release for the Netburner Nano54415 the only A/D example was polled not interrupt driven. So since it was really my day job to write all the divers for the Nano time to write an interrupt driven a/D system for the nano... the first attempt ended up with a single channel A/D conversion interrupt rate at 1.2Mhz seemed like over kill for a pulse whose maximum rate was going to be 500hz or less.... so I turned the A/d clock way down to its minimum value and added in the other 8 A/D channels (only 6 are pined out the chip has 8) and got the rate down to 20Khz or so. The Odometer now was 100% perfectly reliable. (This is the eat your own dog food part.. Ie at Netburner we don't just make modules, we actually try to use them on a regular basis to do things so we understand the customers point of view)
In fact I did a trip around my test building and the dead reckoning was pretty good..
I changed my data logging format so I don't have a copy of the DR trace from my first DR run around the local building, I'm posting one from Sparkfun instead)
Here is a DR plot of two laps around the spark fun building Thursday night.... I drove from where my car was parked in the street out front and went around the building twice...
The DR was almost good enough to to the task all by itself.... I was off about 15ft per lap.
I then started setting up Garbage cans in my drive way and using the LASER range finder to dodge them while staying on course.... I learned that having the laser scanner rotate 90 degrees
to look ahead while also measuring the side range for navigation was problematic. It was fast enough, but the timing of the servo drive and the main command loop were asynchronous enough that I was never really sure where the scanner was pointed and could not properly associate range results with what heading they were on... As I planned on having spare of most everything I then took my spare range finder and mounted it 90 degrees to primary range finder... so now I had two range finders
This was the final physical I/O configuration...
- 3 RC channels from RC receiver in to timer channels.
- 8 RC channels out through a shift register using a timer channel.
- USB serial port for talking to dev system,
- 1 Serial port for receiving data from the IMU (A DIY Drones IMU board)
- 2 Serial port for receiving data from the Laser scanners.
- 1 Serial port receiving data from the Sparkfun Venus GPS.
- 1 A/D for measuring the Odometer.
- 1 DSPI talking to an micro SD card for log data
While using the laser range finder to map barrels in my drive way I accidentally launch the car down the drive way.... still attached to my laptop thus ripping the ethernet jack out of the back.
Ebay to the rescue I got a super rugged lap top from Ebay three days later I had the joy of moving all my tools and code from old laptop to new laptop with out the benefit of ethernet.
I am now very much running out of time. On Tuesday the data from the side laser range finder is really flaky So Wednesday Morning I order another range finder so I have a spare and have it shipped to my hotel room in Boulder.
Wednesday night I go back out to my nearby test building and drive around a few times collecting data. I then spend Wednesday evening fine tuning my data reduction method. For my rocket project I'd written a custom data display application and that was a huge effort. As this seemed to be a simpler project I used a different approach...
The logging was almost identical, IE I logged everything. I stated out by logging to the SD card, but managing file names and when to flush the buffers etc... was a hassle, as I have 64M of available RAM I just logged to RAM and then at the end of the run FTP'ed that data set to the lap top. Quicker and easier than the SD card in this case. For a vehicle that might physically crash such as an airplane or helicopter, or rocket my strategy is to write to the SD card physical sectors directly so the data is always fresh, in this case with the car it did not really apply so I just left it in RAM. (More on the cost of this decision later)
So to reduce the data I wrote a program that would convert the binary log file in to a CSV file and look at it with excel I could do linear plots and scatter plots etc... worked, well but it was slow as to add plots or graphs I had to manually go though the excel interface and add charts...
I eventually figured out how to highlight specific ranges of the data on the scatter plot giving me the ability to tie a specific piece of data to a specific physical location. (See the DR plot above the Pink is a highlighted section of data. Properly viewing multiple rate data is (for me) an unsolved problem.
My Rocket display code worked well, but it was purely linear display, no way to tie the data to a physical location or map. I should probably write something to permanently put in my toolbox for this as its a thorn that keeps poppoing up over and over... either that or learn to wrote VBA code for excel.(yuck).
So Wednesday night I pack up everything, all the spares the car 4 sets of laptop batteries, etc...
I only had one set of Lipos for the car as I was concerned about bringing big Lipos on the plane so the previous week I'd sent 3 sets of Lipos for the car to my hotel in boulder UPS ground.
Soldering tools, a spare everything....
(About 6 pm Wednesday I realize that while I have a spare IMU its got the wrong code in it, so I make a trip to my office where the code load for the IMU Was and reprogram the spare)
I get to bed by midnight or so... and leave the next morning at 5:30 am to catch the plane.
My flight leaves at 7:45 but I'm checking and carrying a lot of strange bits and pieces so I want to allow lots of time. The check in goes fine and the security line is huge, I have my wife waiting in the waiting lot to take the Lipos from me if they don't clear security.
Good thing I had her wait as I go through security it dawns on me I've only gotten three of the four boxes and bags I needed, and I left one in the trunk of the car.... I call her pick that up and wait in line again. Security lets me through.... does not even glance at the 4 laptop batteries or the 3 Large Lipos in my bag.
I arrive in Denver and go to get my rental car.... I have a reservation alas the previous week there was hail in Denver damaging more than half of the rental car fleet. it takes four hours to get my car. There were people in the back of the rental car line that had reservations for a car and weren't going to get a car. There just weren't enough available...
So I drive to boulder, eat some lunch and check into my hotel room. I unpack everything and assemble the car. I drive it around the hotel parking lot to make sure that all of the sensors are working... and head for Spark fun... its clear that they are setting up for the AVC, but there are too many cars in the parking lot to take any useful data to I walk around mentally looking at things and go back to my room. I return to Spark fun around 9 pm that night after all the Spark fun people have left and I'm one of three cars testing at Spark fun.
I drive the car around the building a few times recording all the laser ranges and GPS.
the side facing Laser range finder is still flaky even after swapping out the unit.
So I point the good laser range finder to the right and take two more runs around the building recording that.
I go back to the car make sure I've captured good data and go back to the room.
I get to bed around midnight and get up at 6:15 Friday morning.
I reduce all the data I took into a set of GPS coordinates and a set or range heading and odometer readings. In the last week the AVC project has destroyed my training schedule for the Half Ironman I want to do on my 50th birthday in September , so I use an hour or so to clear my head and get some exercise. At around 9 am I go find the Boulder recreation department public pool and swim about 1.5 miles. Side note swimming is usually a bit hypoxic to begin with, swimming at 5000+ ft of elevation and at fast training pace is hard...)
I go find Lunch and then head back to the room. There is a big empty parking lot that is blocked from traffic near the hotel. I'm not sure what it was for, but it had barriers at the entrance and a bunch of light poles with big concrete bases that were goo barrel dodging simulations.
For practice I drive around four poles and reduce that data to a program that would navigate around the barrels using the barrels and laser range to find the corners... This is working well and then I start testing the barrel dodging code and it all stops working.... it seems that I let the primary battery get too low and the Laser range finders all lost their calibration ARGHHHH! Its now 2 pm and Spark fun has a tour at 3PM, I really want to take the spark fun tour so I drive out to Sparkfun and take the tour. Back at the hotel at 4:30 pm and work on re calibrating the Laser range finders.... I decide to switch the laser range finder to raw count uncalibrated mode and just use raw counts an do the cal myself...
Here I realize what is probably Strategic blunder #0 assuming that other embedded programmers did their job. The Data sheet for the Laser range finder says accurate to +/- 1 Yd no mention of resolution, yet the data print out in Ft mode was DIST:12.3F IE it had lots of resolution, or so I thought... it seems the absolute raw resolution is about 0.5 yd's and the data after the decimal point was 100% meaning less. I had much less resolution that I though I had and finding the arch based on this distance was going to be dicey....
I'm now really worried that the Laser range finding navigation scheme is not going to work at all.
So I do what I should have done weeks ago in San Diego, I write the GPS following code to have a back up plan. I go out to my test parking lot and take GPS coordinates and test this code I successfully drive around and around the parking lot using GPS. It works real well and is more accurate in the open parking lot than it was near my test building.
So at this point its about 9pm and I have a Laser nav program and a Gps nav program set up for Spark fun... both untested on site.... I stop at McDonalds get a burger and a very large Ice tea caffeine supply and proceed to Spark fun.
There are about 5 groups driving around the building with their car.
Between 10PM and midnight I see one group drive their car into the pond twice.....
The first half of the course works well almost immediately....
Finding the first corner is a bit dicey and the spark fun people have put up 2" diameter posts with a 8" diameter base all around the building on the edge of the sidewalk. The laser scanner keeps catching glimpses of these and thinking its too close to the building and as a result it heads for the pond..
My RC kill switch is fast and I save it several times... So I tune the algorithm to take 16 point averages and only use the ranges from the highest 8 or so readings... I also start setting up exclusion zones where the system ignores the laser and just drives on heading... for some fixed distance.
My first around the building state machine had 8 states...
Numbers are odometer counts, reset means set odo back to zero...
/*
DR Version...
//Heading 85 deg Wall follow at 25 ft at 750 arm
//Hold heading till distance jumps by 20 ft
//Then coast 300
//Then turn to 167 Reset Odo
//Then coast 100
//Then follow wall at 28 ft till odo 2000.
//Hold Heading wait till distance jumps to 35 reset odo
//Coast 600
//Then -105 Deg reset odo
//Coast 200
//Wall follow at 42 Arm at odo 1100
//Hold Heading wait for Jump to 52
//Turn to -18 deg Reset Odo Coast 100
//Then wall follow at 20 ft. Arm at 2200
//When distance goes to 30 fto turn to 85 deg
//Coast 100
//Wall follow at 25 ft till odo 1200
//Done.
*/
I have two problems, First the parking lot is not flat, in some places its enough not flat that the car tips enough that the side laser scanner (still being flaky) points at the ground not the building....
So with a bit a cardboard and tape I re point the laser scanner up about 15 degrees.... things get much more consistent.
I keep having a problem with the third turn, no tuning I do seems to make it work. Its now about 2am and I'm at less than peak performance....
So I switch to my GPS code and tune that a little, about 8 tries later the GPS version drives around the building, but in the next three attempts it only works 2/3 of the time. The variance in the low cost GPS is just too much... I save this code as a back up and go back to laser scanner mode...
I immediately see what my issue is I forgot a break in a case statement, so no matter how much I told it to extend out the third turn it just went to the next state turn ... I fix this and a similar copy and paste error two steps later. Next attempt it extends the third turn out way too far and changing this back and now we make it almost all the way around the building.. Fix up the forth turn and we finally make it all the way around the building autonomously with out error. We do this twice more...
I've been runnign after the car all night to see what it is doing. so its speed is set to as fast as I can run... as the night wears on the speed gets set slower so I can keep up.
So I turn the speed back up start the car, watch to make sure it gets past the pond and then I go back the other way to see it come out... no car....
I find it in a bush very far from where it should be....
I try again same result slightly different bush...
I turn the speed back down to where it worked...
Its in a different bush...
So I now run after it.... it works perfectly....
Again I run after it works perfectly.... my knees are starting to hurt from doing wind sprints all night...
I try to launch it and not run after it... strange bush... doesn't work.
Then it dawns on me... when it looses sight of the RC transmitter the RC receiver goes into fail safe mode and stops the car... then when I come looking for it it sees the RC transmitter before I see it and the RC Tx is still in autonomous mode so the car sees this as a manual -> autonomous toggle and starts thinking its at the beginning of the course!!!!! ARghhhh! So I turn the speed up a little and walk ahead of the car as far as I can still retain manual control.... I then start the car and run as fast as I can and it goes around the building... whooo its all working.... then the laser scanner stops working at all!!! Its starting to get light and the first person to show up around 5am with their car says to me early morning or late night.... I make some kind on unintelligible response and one look at me makes him worry that the zombie apocalypse has come to Boulder....
The flaky Laser sensor has felt like bad wiring from the very beginning, if disassemble the housing and hook it directly up to the laptop serial port and power it with an external battery it works every time. I've pulled tugged, and re soldered all the wires from the LASER range finder back to the dev board about 2000 times and can never find a bad connection.....
It starts working again.... but in the process I break the glue joint on the mount....
So I need some glue to re glue the mount and ask the spark fun people (now here setting up) where to find a hardware store and breakfast near by... Or where can I find power to use the Hot Melt Glue gun in my took kit ... They turn on power to the pits area and I hot melt glue the laser scanner mount back together... laser scanners are not both working...
At this point the 4th laptop battery is dead and I go to my car to get the charger only ot realize its in my room at the hotel...
Into the car and drive back to the hotel stopping at the same McDonald's to get breakfast.
Get the charger, change my shirt and try to wipe the grime off my face and hands...
Back to spark fun... its now 8:30 and the contest starts at 9.
I charge the laptop and put fresh Lipos on the car. Sitting on the bench test everything the side laser scanner is not working again... I see a loose wire on the board... The carrier board I'm using only has level translators for one RS-232 serial port. The IMU and GPS are TTL levels, the laser scanners are RS-232 so to get around this I jumper the CTS in line from the one port from the CTS pin to the RX pin on a different serial input (The board has 8 serial ports pinned out) My flaky wiring for the side laser scanner was a bad crimp on the jumper from the CTS to the RX pin... not on the external wiring at all. I fix this and never again have a laser scanner problem for the rest of the day.
I turn the speed down to the speed I can run and the first heat goes reasonably well... we make it 3/4ths of the way around the building the light has changed and now the laser scanner can see through the window on the last turn, it sees this jump in range and thinks the corner of the building has passed and turns into the building....
So I extend the lock out range for corner detection and on the second heat still in slow as I can run mode we make it all the way around in 1:06 We miss the arch so no arch bonus...
(I think this run was good for 4th or 5th place)
This photo was taken by my Friend Ben Brockert (@Wikkit) between the 2nd and third heat.
The last heat... I turn the barrel dodging code back on as last time we just plowed through the barrels... I turn the speed way up and change the code to ignore the RC receiver after going into autonomous mode..... The car hauls down the straight, properly dodges a couple of barrels go straight through the arch and around the corner I run to the other side and it never comes out....
So I walk back and find it in my favorite bush at this point the competition is over and I have a choice... I can carry the 20lb car back to my laptop and download the data log to see what happened... or I can hit reset and manually drive the car back and loose the data log never knowing for sure what happened on the last lap.... I'm tired I press reset....
Sunday, May 06, 2012
Somewhat puzzeling data...
I've now flown the canard finned nose cone, three times...
On the first flight I had the accelerometer sensitivity too high, but the roll sensitivity was fine.
In all hte following plots roll is red, and vertical acceleration is blue. Flown with a CTI L800 blue
We had ever increasing roll as the flight progressed, no evidence of any control
So I changed the acceleration scale and removed the fins, leaving everything else unchanged...
(Slightly differnt motor CTI L820 Skidmark)
The roll was in the same direction and of lower magnitude than the controlled attempt....
Then I re flew with the fins attached and the mechanical throw increased about 50%.
Again with a CTI L820 Skidmark.)
On the first flight I had the accelerometer sensitivity too high, but the roll sensitivity was fine.
In all hte following plots roll is red, and vertical acceleration is blue. Flown with a CTI L800 blue
We had ever increasing roll as the flight progressed, no evidence of any control
So I changed the acceleration scale and removed the fins, leaving everything else unchanged...
(Slightly differnt motor CTI L820 Skidmark)
The roll was in the same direction and of lower magnitude than the controlled attempt....
Then I re flew with the fins attached and the mechanical throw increased about 50%.
Again with a CTI L820 Skidmark.)
This flight almost looks like it worked, ad the roll was controlled during the early boost phase of the flight.
Either I have weird aerodynamics or I need a lot more control authority.
Its possible this last flight worked as designed until it got close to Mach 1 and saw shock wave issues.
The altimeter reported a maximum velocity of 1150 ft/sec or right near Mach 1.
Got to think about this one a bit more...
Subscribe to: Posts (Atom)