avr counter fun

@ Mon Sep 1 23:30:08 PDT 2008
tagged with electronics

Finally started the process of getting my counter off the breadboard and onto a PCB. I am just wiring it by hand, no etching -- but I really need to learn how to use Eagle & etch my own boards, because I am quickly losing my mind/patience/eyesight soldering these wires by hand..

on documentation

@ Wed Aug 29 22:23:21 PDT 2007
tagged with tech

Why write documentation? It's boring! Besides, by the time you've written the documentation, everything has changed! Wahh!

Documenting other people's work may be informative, but it can also make you hate life. Having to write sentences like "I don't know why, but we have to click on the 'blot' and 'zorch' options, otherwise the next screen doesn't display properly" makes one sad.

Documenting your own work, OTOH, can be alot of fun, actually. I find that when I'm documenting things I've done, I generally enjoy it -- it's something of an ego boost, in a strange way. "I wrote this! Here's how it works, here's how you use it, here's where it shines, here's where it needs work."

I embed PODs in my perl (and shell) scripts. I don't do this religiously, but if I know that someone else is going to use it, or even look at it, that's a strong motivator to get some sort of man page put together. When reading other's docs, seeing sample usage helps me quite a bit, so I try to always include an EXAMPLES section. I also find that writing the examples prior to writing code helps me figure out how various options might interact.

Once it's in POD format, I turn it into HTML and publish it for the world (well, people I work with, anyway) to see.

Higher-level architecture documentation usually falls into a wiki -- which is another way of saying I don't spend a whole lot of time on creating printed documentation :-)

Documenting things that change is just an excuse to write a tool to do it for you. :-) At ———, I kept all configuration information in a simple key=value database called the "envstore" -- the envstore contained operationally relevant bits of information like database usernames & passwords, IP addresses, port numbers, and pathnames to where the applications were installed (sensitive information was protected by an ACL :). All the applications retrieved their configurations from the envstore -- none of these values were edited on the target, so the envstore was a pretty close representation of reality.

In addition to making it trivial to relocate applications (application moved from /foo/bar to /somewhere/else/entirely? Change the application root key and redeploy it. Changed servers? Change the IP key and redeploy. Changed passwords? Change the password key and redploy.), the envstore could be mined for all sorts of useful information -- I wrote a CGI to generate documentation from the envstore & let developers, DBAs & business-types the answer the typical questions:

* What environments do we have?

* What's the URL of rev 7 of the order capture application in QA?

* What's the Unix hostname, login & password to the QA environment?

* What's the JDBC URL for rev 8 of the order status application in dev?

* Which hosts are part of the production ATG cluster?

Keep your documentation out of spreadsheets and word documents!

I haven't had a chance to generate network digrams automatically. I'd much rather let a tool generate these kinds of things then maintain it in dia or Visio...

wiggle room

@ Sun Apr 22 22:47:01 PDT 2007
tagged with tech

Tim Bray: Any decent business strategy has to leave enough wiggle room for the discovery of its successor.

this and that

@ Mon Jan 1 23:44:48 PST 2007
tagged with electronics

wow. here we are; we are here. 2007. nice new years dinner at castangas (8-10:30ish, nice & long), champagne there and at afterwards, at home; we drank a little too much.

at some point i should log my cdbaby/hostbaby impressions, since they tend to fluctuate :-)

a few months back, when we went to montana, michelle's mom brought her i-opener to return it to us. i futz with it every so often; cracked the case and exposed the (backwards) IDE connector a month or so ago; been meaning to pick up an laptop-sized IDE cable so I can plug a drive into it to see if i can get linux booted on it. ideally, i could use enough of the on-board flash to get a wifi or bluetooth (?) netboot functional, get X working, and then hang it somewhere as a arty and/or info display screen. with my ereete lectronix skillz i could hook a motion sensor to the parallel port, so i'd know when someone was nearby, so it could wake up and do something in response to movement.

but, i'm not there yet. still need the cable. :-) this weekend I dragged michelle to surplusgizmos.com in hillsborough, where she found a few interesting things to look at while I browsed for a cable, or a cable substitute. While browsing, I noticed they had an ATTiny2313 hooked up to some LEDs and running a cute little blinky routine. thinking/hoping they were the same chip in the minipov2, I asked the guy to google for the minipov and lo and behold, it was! So I got three of 'em ($3 ea), and a an old toshiba laptop for $5 to gut the laptop HD (and whatever else.) Michelle got some headphones and xacto knives.

Got home, figured I needed a halfway decent hack system to play with the ATTiny, so I brought up the old P90 that was flagstaff (now named resistance, as befitting its lectronic experimental purpose), slowly installed etch on the thing, and apt-got all the avr gcc & programmer goodies. Shortly after that I was able to program the minipov to do simple LED blinkyness, and stayed up until 4 AM just fooling around with it.

I actually got the ATTiny 2313-10, which is a 10mhz model. the minipov2 is a -20, but I'm still satisfied :-)

I had little ideas running around in my head while I tried to sleep:

 * water sensor.  with 8 inputs, that's 8 different 
   spots that could be monitored.  since the ports
   are bidirectional, I could have little LEDs attached
   to each sensor, so the corners of the room could blink
   periodically (very important blinky coolness)  .. and
   since the ATTiny2313 has a serial UART (!!@@!) I could
   have a simple little system either send data over
   an RS-232 port to a unix system (ie a line like
       NNNN S0 S1 S2 S3 S4 S5 S6 S7 
	   .. where NNN is an incrementing #, S0..S7 are states (0/1 whether or not
	   there's water), and  just for fun)
   or getting really fancy, be able to interact with the
   device ("give me status of S4")

 * arty led blinky doodad that would slowly pulse different 
   colored LEDs (using PWM or somethin') at slow, peaceful
   rates, but have a microphone to listen for sounds, and
   if it hears a sound, spaz out! (blink all LEDs randomly)
   and when the noise stops, slowly return to a sleepy state.


   upon hearing this idea, michelle said "yes, you could have
   a little device that could listen along with you, and
   spaz out when you do!"  which naturally led into the
   idea of making blinkies with different personalities,
   blinkies that need stimulation to be soothed, blinkies
   that shut off when you make noise and wake up when it's
   quiet, etc.

and tonight for fun i made a cylon-like sweeping LED motion.

these little guys are really cool, they have RAM and flash and a serial port and you can program them in C .. really cool! People have made USB drivers for them, and LCD drivers, and all sorts of interesting little things. I like 'em :-)

pineapple upside-down cake

@ Sat Jul 15 19:21:34 PDT 2006
tagged with food self

mlb is celebrating the 1 year anniversary of her blog with a cake-a-thon. I thought I'd make a pineapple upside down cake. Why a pineapple upside down cake?

* It has the "has the whole world gone mad?" vibe: A cake that you bake upside down and serve right-side-up ? HAS THE WHOLE WORLD GONE MAD?!

* I worked in a bakery at Safeway. I didn't actually do any baking; I washed dishes, swept floors, sliced bread, and stole fitful moments alone in the freezer with boxed chocolate chip cookie dough. One of the few good looking things that came out of the bakery was the pineapple upside-down cake. I was intrigued by the moment of turnover-revalation, and made every effort to be there when it happened; the gooey pan weeping glistening sugary tears onto the cored pineapples and browned cake below. As I boxed them, my face would alight with their radiant beauty, maraschino cherries delightfully centered in the golden fruit. Those sweet cakes, how they made the time in that bakery pass, how oblivious I became to the shouting, the bugs, the occasional dismemberment.. even the tower of dishes seemed to sway gracefully over the sink; even the sweaty floors seemed to soften like a field of alfalfa under the bleach of my mop. Later, when I'd walk home in the hot darkness, the sounds of crickets and cicadas were like a faraway crooning, the sound of pineapples baking in brown sugar.

Requirements:

A can of can of pineapple slices. 8oz, 12oz, whatever.

2 tablespoons of butter (yeah, butter, you know you want it, just try some.)

1/2 cup packed brown sugar

4 maraschino cherries, halved

1 1/2 cups all-purpose flour

2 1/2 teaspoons baking powder

1/3 cups shortening

3/4 cup granulated sugar (we used superfine bakers sugar, which was sublime)

1 egg

1 1/2 teaspoons vanilla

1/4 teaspoon salt

An oven, preheated to 350

A pan of some sort, suitable for baking

A mixer, or a brawny trogdor-like arm

Various measuring implements

Pour off 2/3 of a cup of liquid from the can of pineapple slices into a bowl or measuring cup.

Melt the butter in a pan. Since we had a pyrex baking pan, I put the butter in the pan and the pan in the microwave for about 40 seconds. Stir in the brown sugar and 1 tablespoon of the pineapple liquid from the can. Coat the bottom of the pan evenly.

Next, add an appropriate amount of pineapple slices. Five fit into my pan.

Next, add the maraschino cherries. Don't they look perky?

Now, set that pan aside.

Take the flour, baking power, and 1/4 teaspoon salt and put them in a bowl & combine them.

Next, put your shortening into the mixer and beat it a bit. 30 seconds, 45 seconds, who's counting. Add the sugar. Beat 'till combined. Next, add egg (after cracking it) and vanilla (after sniffing it) and beat for about a minute. Then, alternate between adding the dry flour/baking powder/salt mixture and the cup of pineapple liquid. I was advised/admonished to stop the mixer to add the dry ingredients (taking care to scrape down the bowl) to add the liquid while the mixer was on.

Finally, when all is mixed (don't let it mix too long or the flour will toughen up and you'll wind up making a pineapple-upside-down-bagel-cake), pour/spatula the mixture into the cake pan and spread it evenly.

Bake this fine thing in the oven at 350 degrees for 40 minutes.

After it's done, take it out! Don't let any strangers near it!

Let it rest for 5 minutes, then flip it over and BEHOLD! Scrape any gooey stuff from the pan onto the cake. I scraped along the side of the pan with a butterknife to help ensure the cake was separated from the pan -prior- to flipping it over.

Optionally serve with ice cream / rice dream.

Eat, enjoy, then nap.

restting the counter

@ Mon Jun 12 22:29:08 PDT 2006
tagged with electronics

In the basement with Michelle's machine :-p It's so very cute! I want one of my own. One day. Rev B Black Macbook..

This is my week off before ———. Eek! Picked up my last check from ———.. right into the bank and no doubt will be bill fodder. Gurp.

Revisiting my counter circuit. I found how to 'debounce' (mostly) the connection, the 555 (which I already knew) and a capacitor + resistor in parallel connected in series to the trigger switch. The cap absorbs excess switches (as it charges) and the resistor discharges the cap.. or something like that.

So now I needed a reset button to bring the counter back to zero. The 7490 has a pin that when removed from +5v resets the counter. To build a reset switch, I needed either an (a) normally closed momentary-open switch or (b) a NOT gate.

I found a nice TTL NOT circuit here:

  http://www.play-hookey.com/digital/experiments/ttl_inverter.html

Literally, transistor-transistor. I used 2N222As instead of the 2N4124s. Works like a charm. Now all I need is a tilt switch and I'll be ready to move it off the breadboard and onto a PCB. Wh00t!

After I finish this, I think it would be neat to do the same thing again but this time with a PIC or an Amtel tiny micro, just for fun. I would like to use something I can program easily w/o having to spend $$ on a special programmer. The miniPOV kit I have has a parallel port on it, and is programmable thataway, so ... that might be the way to go. I'd also want to find a way to drive multiple 7-segment LEDs with a single chip.. If there was a "two-BCD-in-to-two-7segment-out" chip, that would work for me ...

my dentist has a band (and i have a new job)

@ Sun May 28 16:11:16 PDT 2006
tagged with self

Was at the dentist again on Thursday for the last bit of tooth work. My dentist mentioned he had a band and I asked him: "have you ever heard of ———?" He had, in fact, and you can listen to them at http://———.com/cd/littlebeirut .. Portland is a such a small town.

Tuesday I went to meet ——— @ ——— for an interviewish/get-to-know-you type of thing. We talked for a few hours about what his plans are for ——— in particular and ——— in general. He was relaxed and friendly. I was nervous as .. a somewhat nervous thing. Should I give up my job at ——— for this? ——— has become such a predictable thing, it'd be hard to leave. Plus, our two new hires would pretty much be on their own after I left. Plus, can I even do the job at ———? I've been so long without r00t, have I forgotten how to be a real sysadmin? I told him I'd sleep on it and give him an answer wednesday. Tuesday night I paced while eating a bowl of asparagus risotto, unsure of myself, my choice, my future. Do I stay at ———, with more of the same frustrations and strictures, or do I burst back into a place not unlike what I left / was taken from me five or six years ago?

Wedesday I fretted. Fret fret fret. Pros:

* Cool place to work .. helping independent musicians, etc.

* Being a real sysadmin! Engineering / architecting / developing / deploying! Ownership of my systems!

* ——— / ——— is a company that's not afraid of technology. No outsourcing!

Cons:

* a big Unknown.

* "Reduced earning potential" -- I'd probably take a drop in salary, or at least bonuses.

* What about benefits?

Wednesday night I said to ———: let's go. Worst case I have to beg for my old job back.

The rest of the night: Terror! Phear! Less salary! What have I done? Aiii!

Thursday: I think this will be cool.

I start at ——— June 19th.

it's frickin' hot in here, mr. bigglesworth.

@ Tue May 16 00:08:54 PDT 2006
tagged with self sysadmin

A beautiful weekend; went for a jog with my dad saturday morning, and then we all went to OMSI and walked around Oaks' Bottom wildlife sanctuary. Sunday I took my parents out for mothers day brunch and then michelle & I planted flowers & vegetables.

Now it's hot. Frickin' hot. My desktop is going to catch on fire at any moment. No breeze. Sweat pooling. It's usually not this hot yet. I don't know what the high was today, but in trying to find out I found this:

 Trojan Implosion Countdown Under Way
 Huge Tower Set To Collapse Early Sunday

 RAINIER, Ore. -- Preparations continue this week for the big 
 implosion of the Trojan Nuclear Plant cooling tower.

 The massive 499-foot tower will implode at 7 a.m. Sunday. Nearby I-5
 will be closed from several minutes before the blast until several
 minutes afterward. It will be visible from Kalama, Wash. for those
 who get there before the closure. However, highway officers and
 Port officers want most people to watch it on television.

 Demolition experts are putting 2,500 pounds of dynamite into
 hundreds of pre-drilled holes.

 The tower has been a Northwest landmark for more than three
 decades. It was closed in 1993.

 Experts say there is no danger of radiation release. The cooling
 tower is not where nuclear energy was produced. That happened in
 the nearby, and much smaller, dome-shaped reactor building.

 5/15/2006

Yeek!

----

Been exchanging emails w/ ——— @ ——— for a bit, he's interested in hiring and i'm interested something new.

——— liked me, but I don't want to (a) relocate or (b) do the 80 mile commute..

----

Downloaded the OpenSolaris ISOs & burned 'em. Had some problems getting it to boot on my test box, kept getting a "consconfig : no screen found". My video card is an old 512k ISA card, I wouldn't think it would matter, but perhaps it just got confuzed while probing it; it didn't expect hardware that old, or only knew to probe for PCI video. I was able to get into the installer on michelle's old E-machine w/ integrated video, so I'm pretty sure I just need to put in a slightly more modern video card.

I'd really like to start playing with ZFS in particular and Solaris 10 in general. if ZFS is as gr00vy as it's claimed, it would be hard for me justify to spend time futzing linux w/ LVM and ext3. I could put Xen on my big 1r0n server, and run a Solaris 10 image and a linux image side by side, and let the solaris image be the file server... mmm..

I bought a 4-port SATA card and two 250gb drives. Gonna move barcelona:/export/archive from the 100% full 2-disk 60gb raid5 array to a 25% full 250gb RAID1(0) array. But first I will try the card in the test box to see if it works under Solaris 10 ..!

----

gack. spamassassin is catching SIGSEGV and dying.. gurgle..

 grsec: From 172.16.16.5: signal 11 sent to /jsi/soe/dist/linux/debian-3.0/i686/perl-modules/1.0/bin/spamd[spamd:7591] uid/euid:0/0 gid/egid:0/0, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0
 grsec: From 172.16.16.5: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /jsi/soe/dist/linux/debian-3.0/i686/perl-modules/1.0/bin/spamd[spamd:7591] uid/euid:0/0 gid/egid:0/0, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0

foo.

five by five

@ Fri May 5 20:56:43 PDT 2006
tagged with self

Beautiful day today! Took the day off. Went for a bike ride, first of the year. Took a break at the half-way point, at the park by the sellwood bridge. Laid on the grass and watched the river, the dogs, the grass, the cute little green bugs that like to walk around on my pale skin. They get trapped in the forest of my massively hairy, ape-like arms. I am gentle with them, for they are not for long in this world (at least on my timeline.)

Had another conversation w/ ——— this morning, w/ the manager of the group. Not crazy about relocating.

Dinner was pok pok. Mmmm, tasty pork

My second electronics project, the digital counter, is coming along nicely. howstuffworks.com had a "how digital clocks work" page -- a decade counter, a BCD -> 7 segment display driver, and a 555 timer IC to deliver pulses. Built one counter on my existing breadboard, but ran out of space and got another one, a little nicer (& bigger) than the one from radio shack.

I drained a 9v battery pretty quickly getting it working, got some 7805 voltage regulators and hooked it up to a wall wart.. the 7805 gets frickin hot! I could smell it cooking. I added 3 1000 ohm resistors in parallel to the source supply and it's running a little cooler now... at least there's no smell :)

I need to figure out how to make a switch (mercury or something) to deliver a clean pulse. there is a circuit called a monostable multivibrator that can do this. a 555 can be configured as a monostable multivibrator. yay! now i just have to breadboard it, and then figure out how to lay it out on the circuit board. There will be quite a few wires :) maybe I should try to etch a PCB, or have a PCB made for me.. would certainly be cleaner. The whole layout & mounting will be tricky!

my first parallel port project

@ Sat Apr 15 20:03:22 PST 2006
tagged with electronics

I have a DSL modem that periodically stops modeming. I've upgraded the firmware to the latest rev(s), double-checked power connections, and prayed, but to no avail. A power cycle solves the problem, but eventually it fails again. There doesn't seem to be a pattern to the failure, either; hot weather, cold weather, it dies.

It would be nice to be able to powercycle the DSL modem when it failed. X-10 was the first obvious choice, but my previous experiences with the "bottlerocket" stuff showed that it was't all that reliable. Besides, it wasn't much as trying to do it yourself..

I wanted to learn how to drive things from the parallel port. So I first figured out how to turn an LED on and off.

Next: drive a relay. I would have liked to have taken power from the parallel port, but it was not reccomended. Used an external 9v battery to open & close it.

Interface to the parallel port: had some DB9 <-> DB25 cables with RJ11 in between. Found a DB-9 female connector for the controller side.

Finally, the hookup: two female power plugs, and a male <-> male connector. Relay between the power plugs. Power source on one side, and male-male cable to the modem on the other.

Then: what to put it in? Found an 8mm tape case.

i hate myself for working for you

@ Tue Apr 11 15:09:54 PDT 2006
tagged with work

The ——— operations team must justify its access to its servers in production. That's right -- the team that's responsible for the production environment isn't allowed to login to the production boxes.

When asked for a document describing what access is required, I said:

* The ability to execute programs out of the following directories:

  /bin /usr/bin /sbin /usr/sbin /lc /cust

* The ability to access any shared library that is dependent on programs in the above directories.

* The ability to use any tool in the above directories to read and/or parse

files within the following directories:

  /var/log /var/adm /cust

!@#

Reponsibility without Authority. Good times.

DS-3 asynchronously

@ Mon Mar 13 20:03:22 PST 2006
tagged with work network

Ok, here's why we couldn't pass nulls:

  The trouble was isolated to an ——— controlled SMX OC12
  fiber mux on the ——— ——— campus that extends the circuit between
  campus buildings. The Option 1 circuit goes through the SMX mux that was
  not optioned to ignore excessive zeroes which is usually not a problem
  unless the customer's application is pushing excessive 0s across the
  circuit. In this case the customer's SAP application was padding 0s
  which resulted in a large string of 0s being transmitted across the
  connection and the fiber mux was not able to maintain synchronization
  when the string of 0s were detected. This resulted in the packet drops
  the customer was reporting.

  ——— actually has two different versions of fiber mux that they
  use when transporting connections between buildings on their fiber ring.
  They use what is called an Alcatel SMX mux (must be optioned to ignore
  excessive 0s if the customer traffic pattern generates this condition)
  and an Alcatel SM fiber mux which does not have the ability to be
  optioned to ignore excessive 0s. In ———'s case, the primary
  SMX fiber mux was already optioned to ignore excessive 0s when they 
  are presented and the secondary fiber mux was not set to ignore the 
  excessive 0s and the Engineer had to make the configuration change to correct
  that. 
  

I don't know enough about the magic telco protocols to understand why this would affect synchronization.

* If it's synchronous, there's clocks at either end that are synchronized. Data spews forth in predictable (clocked) chunks from one side, and the other side reassembles the chunks according to the clock.

* If it's asynchronous, the byte should be encapsulated by a bit that indicates "start of data."

I thought these protocols were all sync, but maybe they're not. Spooky.

new glasses

@ Sat Mar 11 15:44:33 PST 2006
tagged with self

I got new glasses today. Damn, I'm smoov..

.. I do have a big head, don't I ...

it's the DS-3, not me.

@ Sun Mar 6 20:42:19 PST 2006
tagged with work network

The DS-3 can't pass zeroes. Why? Who knows.

This evening I had the circuit provider engineer route traffic from my Linux desktop across the DS-3. I used the perl doodad to generate the problematic SAP traffic from my desktop, and wrote another perl doodad to echo the traffic back to me. In essence it was a ping; send data to the server, and the server sends it right back 'atcha.

The ping failed. Failed! This means the problem isn't in the SAP server (hooray; troubleshooting that would have sucked bigtime.) I start chopping the packet down, trying to figure out which sequence of bytes are not passing. Finally, I am left with about 96 nulls. They don't pass. I mention this to the engineer. He says "Huh! that sounds familiar!" and a few seconds later says "I'm able to confirm this." He used ping on one of the routers to send a few dozen null bytes to the other router, and confirms they don't pass. I play with 'ping -p' from my Linux machine and confirm as well. Hooray! Now that the circuit provider can replicate the problem, we can stop pointing fingers and get things going.

SAP, DS-3, you, and me.

@ Sun Mar 5 23:43:28 PST 2006
tagged with work network

Since we moved datacenters, we've been having problems with the circuits that connect the campus to our machines (all the way in Texas.) The vendor requested a window in which they could do some 'invasive' testing. During this time we would fail over to the secondary circuit and so no traffic should be affected.

Or so we thought..

Right after we switched over, I started getting alerts from Nagios and Gomerz, and started seeing lots of SAP timeout errors in the app logs. uh-oh. We have a little standalone JCO/RFC client 'ping' that we use to validate basic connectivity back to SAP, and even it was failing. Boo. I play for a little bit, seeing how far we get.

  1. Can we ping the SAP server?  Check.  
  2. Can we establish a TCP 3-way handshake to the SAP server?  Yup.  
  3. Does the client start sending data to the SAP server?  Uh-huh.  
  4. Does the server send data back to the SAP client?  Mmm-hmm.
  5. Does the client send more data to the SAP server?  You know it to be true.
  6. Does the server respond back to this?  No!

For some reason SAP did not play nice with the secondary circuit. Since we didn't anticipate this, we called off the testing window and regrouped to try to isolate the problem. I was able to capture one network trace, as well as a truss from the JCO/RFC ping utility.

I spent a little time looking at what the ping utility did, and tried to figure out where it was breaking. After playing around a bit, I found that it fell over when it requested a list of services the SAP gateway offers.

[ Aside: I am no SAP expert, but here's what I do know: the client makes a TCP connection to the gateway, and the gateway tells the client which application server to connect to. The client then disconnects from the gateway and connects to the app server. This is SAP's method of load balancing. It is roughly analagous to a rendezvous-server, or the Unix portmapper. ]

I was able to cobble together a perl script to get a list of IP addresses out of the gateway --

 jwa@nimue:~/work/cvs/hacks/sap$ ./talk-to-gw 
 writing msg1 [114 bytes]
 response length: 110
  **MESSAGE**-                                      MSG_SERVER                              
 got 110 bytes
 writing msg2 [254 bytes]
 response length: 1290

 parse
 Name:  [**MESSAGE**--??]
 Type:  0x0 (0)
 Name:  [B2B_GRP             LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.251.197
 Name:  [BILLER_DIRECT       LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.3.118
 Name:  [DE1-APP             LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.251.197
 Name:  [DP_APP              LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.251.197
 Name:  [JCO_ORDER_STAT      LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.251.197
 Name:  [Long_Runner         LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.251.197
 Name:  [MAIL                LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.251.197
 Name:  [FAV_COMPUTE_TIME    FAV_COMPUT]
 Type:  0x54 (84)
 Name:  [FAV_COMPUTE_SERVER  camsapd1_D]
 Type:  0x44 (68)
 Name:  [SPACE               LG_EYECAT]
 Type:  0x1 (1)
 IP: ———.3.118
 Name:  []
 Type:  0x0 (0)

It was at the point that the client sends the second message of 254 bytes ('msg2' above) to the server that the connection to the gateway hung, and the client eventually timed out. It looked like the packet (& subsequent retries) was either dropped by the network, or not properly read by the server.

Therefore, two speculations:

1. The secondary circuit modifies the characteristics of the packet (ie, fragmentation, or MTU shrinkage, or whatever) such that the 254 byte packet is delevered in two pieces. The SAP gateway expects exactly 254 bytes of data in a single read(), but since it gets (say) 128 bytes of data, it does the Wrong Thing.

2. The secondary circuit is dropping packets that have a certain byte pattern. (like pinging a dialup user with a packet containing '+++'; their stack echoes it back, and the modem sees it as '+++' originating from the client, and goes into command mode & breaks the users' dialup session.)

The circuit provider doesn't think they're dropping packets, and suspect it's an app issue.

Soo.. we spend the next week figuring out how to replicate the problem w/o affecting production. We decide to route traffic between the SAP QA system and our QA system across the secondary circuit. (Much too much time is spent on getting approval from any remotely interested party.) We decide to place no less than four sniffers on the network-- one close to the SAP QA system, one at the edge of our network (where our network meets the circuit), one at the edge of our hosting provider's network (where their network meets the circuit), and one right off our QA network.

Saturday comes. The circuit provider routes traffic (it's neat to see the traceroutes change) across the secondary circuit. And, presto! we can replicate the problem... even with my little perl script :-) We capture sniffer traces and spend ~2h analyzing them, this time with the assistance of our hosting provider's network folks. We see a 254 byte packet leaving the edge of the hosting provider's network, but we dont'see it on our edge of the network. We see the TCP stack on the client end get lonely and issue retransmits, but we don't see those retransmissions on our edge.

It seems pretty clear that the packet is getting lost in "the cloud". But why? No one knows. The people that might be able to help aren't on the call, and we agree to pick things up on Monday.

Afterwards, I went for a gentle jog around work. My knee is still acting up, though, so I walked on the downhill parts to minimize trauma. While I could feel it threatening to hurt, it felt mostly OK up until I felt a strange pain in the back of my knee. I walked for a bit after that, fortunately it was near the end. Going down the stairs later that evening I could definitely feel it. Gah.