Monday, February 22, 2016

Thank You

Back in the day, Joel Spolsky had a very influential tech blog, and one of the pieces he wrote described the kind of software developer he liked to hire, one who was "Smart, and gets things done." He later turned it into a book ( Steve Yegge, who was also a very influential blogger in the oughties, wrote a followup, in which he tackled the problem of how you find and hire developers who are smarter than you. Given the handicaps of human psychology, how do you even recognize what you're looking at? His rubric for identifying these people (flipping Spolsky's) was "Done, and gets things smart". That is, this legendary "10X" developer was the sort who wouldn't just get done the stuff that needed to be done, but would actually anticipate what needed to be done. When you asked them to add a new feature, they'd respond that it was already done, or that they'd just need a few minutes, because they'd built things in such a way that adding your feature that you just thought of would be trivial. They wouldn't just finish projects, they'd make everything better—they'd create code that other developers could easily build upon. Essentially, they'd make everyone around them more effective as well.

I've been thinking a lot about this over the last few months, as I've worked on finishing a project started by Sebastian Rahtz: integrating support for the new "Pure ODD" syntax into the TEI Stylesheets. The idea is to have a TEI syntax for describing the content an element can have, rather than falling back on embedded RelaxNG. Lou Burnard has written about it here: Sebastian wrote the XSLT Stylesheets and the supporting infrastructure which are both the reference implementation for publishing TEI and the primary mechanism by which the TEI Guidelines themselves are published. And they are the basis of TEI schema generation as well. So if you use TEI at all, you have Sebastian to thank.

Picking up after Sebastian's retirement last year has been a tough job. It was immediately obvious to me just how much he had done, and had been doing for the TEI all along. When Gabriel Bodard described to me how the TEI Council worked, after I was elected for the first time, he said something like: "There'll be a bunch of people arguing about how to implement a feature, or even whether it can be done, and then Sebastian will pipe up from the corner and say 'Oh, I just did it while you were talking.'" You only have to look at the contributors pages for both the TEI and the Stylesheets to see that Sebastian was indeed operating at a 10X level. Quietly, without making any fuss about it, he's been making the TEI work for many years.

The contributions of software developers are often easily overlooked. We only notice when things don't work, not when everything goes smoothly, because that's what's supposed to happen, isn't it? Even in Digital Humanities, which you'd expect to be self-aware about this sort of thing, the intellectual contributions of software developers can often be swept under the rug. So I want to go on record, shouting a loud THANK YOU to Sebastian for doing so much and for making the TEI infrastructure smart.

UPDATE 2016-3-16

I heard the sad news last night that Sebastian passed away yesterday on the Ides of March. We are much diminished by his loss.

Friday, October 25, 2013

DH Data Talk

Last night I was on a panel organized by Duke Libraries' Digital Scholarship group. The panelists each gave some brief remarks and then we had what I thought was a really productive and interesting discussion. The following are my own remarks, with links to my slides (opens a new tab). In my notes, //slide// means click forward (not always to a new slide, maybe just a fragment).
This is me, and I work //slide// for this outfit. I'm going to talk just a little about a an old project and a new one, and not really give any details about either, but surface a couple of problems that I hope will be fodder for discussion. //slide// The old project is and publishes all kinds of data about ancient documents mostly written in ink on papyrus. The new one, Integrating Digital Epigraphies (IDEs), is about doing much the same thing for ancient documents mostly incised on stone.
If I had to characterize (most of) the work I'm doing right now, I'd say I'm working on detecting and making machine-actionable the scholarly links and networks embedded in a variety of related projects, with data sources including plain text, XML, Relational Databases, web services, and images. These encompass critical editions of texts (often in large corpora), bibliography, citations in books and articles, images posted on Flickr, and databases of texts. You could think of what I'm doing as recognizing patterns and then converting those into actual links; building a scaffold for the digital representation of networks of scholarship. This is hard work. //slide// It's hard because while superficial patterns are easy to detect, //slide// without access to the system of thought underlying those patterns (and computers can't do that yet—maybe never), those patterns are really just proxies kicked up by the underlying system. They don't themselves have meaning, but they're all you have to hold on to. //slide// Our brains (with some prior training) are very good at navigating this kind of mess, but digital systems require explicit instructions //slide// —though granted, you can sometimes use machine learning techniques to generate those.
When I say I'm working on making scholarly networks machine actionable, I'm talking about encoding as digital relations the graph of references embedded in these books, articles and corpora, and in the metadata of digital images. There are various ways one might do this, and the one we're most deeply into right now is called //slide// RDF. RDF models knowledge as a set of simple statements in the form Subject, Predicate, Object. //slide// So A cites B, for example. RDF is a web technology, so all three of these elements may be URIs that you could open in a web browser, //slide// and if you use URIs in RDF, then the object of one statement can be the subject of another, and so on. //slide// So you can use it to model logical chains of knowledge. Now notice that these statements are axioms. You can't qualify them, at least not in a fine-grained way. So this works great in a closed system (, where we get to decide what the facts are; it's going to be much more problematic in IDEs, where we'll be coordinating data from at least half a dozen partners. Partners who may not agree on everything. //slide// What I've got is the same problem from a different angle—I need to model a big pile of opinion but all I have to do it with are facts.
Part of the solution to these problems has to be about learning how to make the insertion of machine-actionable links and facts (or at least assertions), part of—that is, a side-effect of—the normal processes of resource creation and curation. But it also has to be about building systems that can cope with ambiguity and opinion.

Wednesday, September 11, 2013

Outside the tent

Yesterday was a bad day. I’m chasing a messed-up software problem whose main symptom is the application consuming all available memory and then falling over without leaving a useful stacktrace. Steve Ramsay quit Twitter. A colleague I have huge respect for is leaving a project that’s foundational and is going to be parked because of it (that and the lack of funding). This all sucks. As I said on Twitter, it feels like we’ve hit a tipping point. I think DH has moved on and left a bunch of us behind. I have to start this off by saying that I really have nothing to complain about, even if some of this sounds like whining. I love my job, my colleagues, and I’m doing my best to get over being a member of a Carolina family working at Duke :-). I’m also thinking about these things a lot in the run up to Speaking in Code.

For some time now I’ve been feeling uneasy about how I should present myself and my work. A few years ago, I’d have confidently said I work on Digital Humanities projects. Before that, I was into Humanities Computing. But now? I’m not sure what I do is really DH any more. I suspect the DH community is no longer interested in the same things as people like me, who write software to enable humanistic inquiry and also like to think (and when possible write and teach) about how that software instantiates ideas about the data involved in humanistic inquiry. On one level, this is fine. Time, and academic fashion, marches on. It is a little embarrassing though given that I’m a “Senior Digital Humanities Programmer”.

Moreover, the field of “programming” daily spews forth fresh examples of unbelievable, poisonous, misogyny and seems largely incapable of recognizing what a shitty situation its in because of it.

The tech industry is in moral crisis. We live in a dystopian, panoptic geek revenge fantasy infested by absurd beliefs in meritocracy, full of entrenched inequalities, focused on white upper-class problems, inherently hostile to minorities, rife with blatant sexism and generally incapable of reaching anyone beyond early adopter audiences of people just like us. (from

I think communities who fight against this kind of oppression, like #DHPoco, for example, are where DH is going. But while I completely support them and think they’re doing good, important work, I feel a great lack of confidence that I can participate in any meaningful way in those conversations, both because of the professional baggage I bring with me and because they’re doing a different kind of DH. I don’t really see a category for the kinds of things I write about on DHThis or DHNow, for example.

This is great stuff, but it’s also not going to be a venue for me wittering on about Digital Classics or text encoding. It could be my impostor syndrome kicking in, but I really doubt they’re interested.

It does seem like a side-effect of the shift toward a more theoretical DH is an environment less welcoming to participation by “staff”. It’s paradoxical that the opening up of DH also comes with a reversion to the old academic hierarchies. I’m constantly amazed at how resilient human insitutions are.

If Digital Humanities isn’t really what I do, and if Programmer comes with a load of toxic slime attached to it, perhaps “Senior” is all I have left. Of course, in programmer terms, “senior” doesn’t really mean “has many years of experience”, it’s code for “actually knows how to program”. You see ads for senior programmers with 2-3 years of experience all the time. By that standard, I’m not Senior, I’m Ancient. Job titles are something that come attached to staff, and they are terrible, constricting things.

I don’t think that what I and many of my colleagues do has become useless, even if we no longer fit the DH label. It still seems important to do that work. Maybe we’re back to doing Humanities Computing. I do think we’re mostly better off because Digital Humanities happened, but maybe we have to say goodbye to it as it heads off to new horizons and get back to doing the hard work that needs to be done in a Humanities that’s at least more open to digital approaches than it used to be. What I’m left wondering is where the place of the developer (and, for that matter other DH collaborators) is in DH if DH is now the establishment and looks structurally pretty much like the old establishment did.

Is digital humanities development a commodity? Are DH developers interchangeable? Should we be? Programming in industry is typically regarded as a commodity. Programmers are in a weird position, both providers of indispensable value, and held at arm’s length. The problem businesses have is how to harness a resource that is essentially creative and therefore very subject to human inconsistency. It’s hard to find good programmers, and hard to filter for programming talent. Programmers get burned out, bored, pissed off, distracted. Best to keep a big pool of them and rotate them out when they become unreliable or too expensive or replace them when they leave. Comparisons to graduate students and adjunct faculty may not escape the reader, though at least programmers are usually better-compensated. Academia has a slightly different programmer problem: it’s really hard to find good DH programmers and staffing up just for a project may be completely impossible. The only solution I see is to treat it as analogous to hiring faculty: you have to identify good people and recruit them and train people you’d want to hire. You also have to give them a fair amount of autonomy—to deal with them as people rather than commodities. What you can’t count on doing is retaining them as contingent labor on soft money. But here we’re back around to the faculty/staff problem: the institutions mostly only deal with tenure-track faculty in this way. Libraries seem to be the only academic institutions capable of addressing the problem at all. But they’re also the insitutions most likely to come under financial pressure and they have other things to worry about. It’s not fair to expect them to come riding over the hill.

The ideal would situation would be if there existed positions to which experts could be recruited who had sufficient autonomy to deal with faculty on their own level (this essentially means being able to say ‘no’), who might or might not have advanced degrees, who might teach and/or publish, but wouldn’t have either as their primary focus. They might be librarians, or research faculty, or something else we haven’t named yet. All of this would cost money though. What’s the alternative? Outsourcing? Be prepared to spend all your grant money paying industry rates. Grad Students? Many are very talented and have the right skills, but will they be willing to risk sacrificing the chance of a faculty career by dedicating themselves to your project? Will your project be maintainable when they move on? Mia Ridge, in her twitter feed, reminds me that in England there exist people called “Research Software Engineers”.

There are worse labels, but it sounds like they have exactly the same set of problems I’m talking about here.

Monday, July 15, 2013

Missing DH

I'm watching the tweets from #dh2013 starting to roll in and feeling kind of sad (and, let's be honest, left out) not to be there. Conference attendance has been hard the last few years because I didn't have any travel funding in my old job. So I've tended only to go to conferences close to home or where I could get grant funding to pay for them.

It's also quite hard sometimes to decide what conferences to go to. On a self-funded basis, I can manage about one a year. So deciding which one can be hard. I'm a technologist working in a library, on digital humanities projects, with a focus on markup technologies and on ancient studies. So my list is something like:
  • DH
  • JCDL
  • One of many language-focused conferences
  • The TEI annual meeting
  • Balisage
I could also make a case for conferences in my home discipline, Classics, but I haven't been to the APA annual meeting in over a decade. Now that the Digital Classics Association exists, that might change.

I tend to cycle through the list above. Last year I went to the TEI meeting, the year before, I went to Clojure/conj and DH (because a grant paid). The year before that, I went to Balisage, which is an absolutely fabulous conference if you're a markup geek like me (seriously, go if you get the chance).

DH is a nice compromise though, because you get a bit of everything. It's also attended by a whole bunch of my friends, and people I'd very much like to become friends with. I didn't bother submitting a proposal for this year, because my job situation was very much up in the air at the time, and indeed, I started working at DC3 just a couple of weeks ago. DH 2013 would have been unfeasible for all kinds of reasons, but I'm still bummed out not to be there. Have a great time y'all. I'll be following from a distance.

Wednesday, February 06, 2013

First Contact

It seems like I've had many versions of this conversation in the last few months, as new projects begin to ramp up:
Client: I want to do something cool to publish my work.

Developer: OK. Tell me what you'd like to do.

Client: Um. I need you to to tell me what's possible, so I can tell you what I want.

Developer: We can do pretty much anything. I need you to tell me what you want so I can figure out how to make it.
Almost every introductory meeting with a client/customer starts out this way. There's a kind of negotiation period where we figure out how to speak each other's language, often by drawing crude pictures. We look at things and decide how to describe them in a way we both understand. We wave our hands in the air and sometimes get annoyed that the other person is being so dense.

It's crucially important not to short-circuit this process though. You and your client likely have vastly different understandings of what can be done, how hard it is to do what needs to be done, and even whether it's worth doing. The initial negotiation sets the tone for the rest of the relationship. If you hurry through it, and let things progress while there are still major misunderstandings in the air, Bad Things will certainly happen. Like:
Client: This isn't what I wanted at all!

Developer: But I built exactly what you asked for!

Thursday, April 26, 2012

You did _what_? Form-based XML editing

My most recent project has been building an editing system for APIS (the Advanced Papyrological Information System—I know, I know) data as part of the Papyrological Editor (PE). APIS records are now TEI files. They started out using a text-based format that was MARC-inspired (fields, not structure), but I converted them to TEI a while back to bring them in line with the other data. HGV records (also TEI) already had an editing system in the PE, so it seemed like the task of adding an APIS editor, which would be a simpler beast, wouldn't be that hard, just a matter of extending the existing one. In fact, it was an awful struggle.

Partly this is my fault. The PE is a Rails application, running on JRuby, and my Rails knowledge was pretty rusty. I loved it when I was working on it full time a few years ago, but coming back to it now, and working on an application someone else built, I found it obtuse and mentally exhausting. I had a hard time keeping in my head the different places business logic might be found, and figuring out where to graft on the pieces of the new editor was a continual fight against coder's block. The HGV editor relies on a YAML-based mapping file that documents where the editable information in an HGV document is to be found. This mapping is used to build a data structure that is used by the View to populate an editing form, and upon form submission, the process is reversed.

It's not at all unlike what XForms does, and in fact I was repeatedly saying to myself "Why didn't they just use XForms?" I got annoyed enough that I took some time to look at what it would take to just plug XForms into the application and use that for the APIS editor. The reluctant conclusion I came to was that there just aren't any XForms frameworks out there that I could do this with. And the XForms tutorials I was looking at didn't deal with data that looked like mine at all. TEI is flexible, but not all that regular, and every example I saw dealt with very regular XML data. Moreover, the only implementation I could find that wasn't a server-side framework (and I wasn't going to install another web service just to handle one form) is XSLTForms. The latter is impressive, but relies on in-browser XSLT transforms, which is fine if you have control of the whole page, but inconvenient for me, because I've already got a framework producing the shell of my pages. I just wanted something that would plug into what I already had. A bit sadder but wiser, I decided the team who built the HGV editor had done what they had to given what they had to work with.

Then, about a month ago, I got sick. Like bedridden sick. I actually took 3 days off work, which is probably the most sick leave I've taken since my children were born. While I was lying in bed waiting for the antibiotics to kick in, I got bored and thought I'd have a crack at rewriting the APIS editor. In Javascript. There was already a perfectly good pure-XML editing view, where you can open your document, make changes, and save it back (having it validated etc. on the server), so why not build something that could load the XML into a form in the browser, and then rebuild the XML with the form's contents and post it back? Doesn't sound that hard. And so that's what I did. I should say, this seemed like a potentially very silly thing to do, adding another, non-standard, editing method that duplicates the functionality of an existing standard to a system that already has a working method. I'd spent a lot of time on figuring out how to refactor the existing editor because that seemed like the Right Thing To Do. Being sick, bored, and off the clock gave me the scope to play around a bit.

Let me talk a bit more about the constraints of the project. Data is stored in XML files in a Git repository. This means that not only do I want my edits to be sending back good XML, I want it to look as much like the XML I got in the first place, with the same formatting, indentation, etc., so that Git can make sensible judgements about what has actually changed. I might want some data extracted from the XML to be processed a bit before it's put into the form, and vice versa. For example, I have dates with @when or @notBefore/@notAfter attributes that have values in the form YYYY-MM-DD, but I want my form to have separate Year, Month, and Day fields. Mixed (though only lightly mixed) content is possible in some elements. I need my form to be able to insert elements that may not be there at all in the source. I need it to deal with repeating data. I need it to deal with structured data (i.e. an XML fragment containing several pieces of data should be treated as a unit). And of course, it needs to be able to put data into the XML document in such a way that it will validate, so the order of inserted elements matters. Moreover, I need to build the form the way the framework expects it to be built, as a Rails View.

So the tool needs to know about an instance document:

  1. how to get data out of the XML, maybe running it through a conversion function on the way out
  2. how to map that data to one or a group of form elements, so there needs to be a naming convention
  3. how to deal with repeating elements or groups of elements
  4. how to structure the data coming out of the form, possibly running it through a function, and being able to deal with optional sub-structures
  5. how to put the form data into the correct place in the XML, so some knowledge of the content model is necessary
  6. how to add missing ancestor elements (maybe with attributes) for new form data
Basically, it needs to be able to deal with XML as it occurs in the wild: nasty, brutish, and (one hopes) short.

Form-based XML editing is one of those things that sounds fairly easy, but is in fact fraught with complications. It's easy to get data out of XML (XPath!), and it's easy to manipulate an XML DOM, removing and adding elements, attributes, and text nodes. But it's actually quite hard to get everything right, and to make the XML's formatting stay consistent. In my next post, I'll talk about how I did it.

Tuesday, March 20, 2012

How to Apologize

The latest regrettable spasm of sexism in the programming world played out this afternoon, as a company called Sqoot's announcement of a hackathon caused said event to implode before it ever began by including the infuriating and insensitive line under "perks" [Update: just to be clear, in the context of the original page, it was clear that the presence of women serving beer was one of the perks for attendees]:
Women: Need another beer? Let one of our friendly (female) event staff get that for you.
Gag. Sqoot fairly quickly realized they had walked into a buzzsaw, as lots of people called them on it, and their sponsors started pulling their support. It's rather nice to see that kind of quick, public reaction. Cloudmine's blog post about it particularly impressed me. Squoot issued an apology fairly swiftly, which I quote below:
Sqoot is hosting an API Jam in Boston at the end of March. One of the perks we (not our sponsors) listed on the event page was:

“Women: Need another beer? Let one of our friendly (female) event staff get that for you.”

While we thought this was a fun, harmless comment poking fun at the fact that hack-a-thons are typically male-dominated, others were offended. That was not our intention and thus we changed it.

We’re really sorry,

Avand & Mo
This didn't do much for a lot of people, but it got me thinking about apologies in tech in general, since they are actually crucial moments in the interaction between you and your customers/audience. When I worked at Lulu, Bob Young used to say that whenever you screw up, it's actually a tremendous opportunity to win a customer's loyalty by making it right. This applies both to small screwups (a customer's order never made it) and large ones (you did something that made lots of people mad). It strikes me that in this day and age, when the "non-apology" has become so frequent, people may actually not realize when it isn't appropriate to use conditional or evasive language in apologies. It's one thing if you're worried about being sued and can't admit culpability, or if you're someone like Rush Limbaugh, who's presumably concerned about appearing to back down in front of his audience. But if you're actually intent on repairing the damage done to your relationship with your customer or your audience, you need to be able to apologize properly.

So what are the elements of a good apology?

  1. I hear you.
  2. I am truly sorry.
  3. (semi-optional, depending on what happened) This is what went wrong.
  4. I am doing x to make sure this doesn't happen again and y to make it right with you.
  5. Thank you. I appreciate the feedback.
#1 is crucial. The person or group you're addressing has to know that you've heard their complaint and understand it. Apologies that lack this element sound cold and disconnected. And this is the main problem with Sqoot's "others were offended."  They aren't speaking to the people they offended. This is just guaranteed to further piss people off.

#2 should be unconditional. Not "I'm sorry if you were offended." Indeed, if you find yourself pushing the focus onto the people whom you pissed off at all, you may be sliding into non-apology territory. This isn't about them—they're mad because you made them mad. Note that a good apology is not defensive, and does not attempt to shift the blame, even if that blame belongs to an employee whom you've just fired.  If you did that, it's part of #4, the "how I'm fixing it" part, not the "I'm sorry" part. Don't try to save face in a genuine apology. Indicating that you meant no harm is fine, but if you're apologizing, it means you caused harm regardless of your intent.

#3 is a bit more tricky. People want to know how this could have happened, but it doesn't do to dwell on it too much, and this is another mistake Sqoot makes. They probably shouldn't quote the line that made everyone mad (it will make the readers mad all over again). It would have been enough to say they put something stupid and sexist into an event page which they now regret. On the other hand, you do have to acknowledge what happened and not look like you're trying to dodge it. So don't go into excruciating detail about what went wrong with a customer's order, for example. "I'm afraid you found a bug in our shopping cart" is probably enough detail. Sqoot's apology does this really badly: they explain exactly what they did, how it happened (we thought it was funny, because we're aware that these things are mostly male), and then contrast the "others" (who lack their sense of humor) who were offended. Explaining how you messed up does not mean defending yourself, and defending yourself in an apology must be handled delicately, or you look like an ass.

#4 Fix it, if you can. "We're refunding your order immediately and giving you a coupon", "I shall be entering rehab tomorrow morning", "We're donating $$ to x charity".

#5 Reconnect. When you screw up, people are paying very close attention to you, and it's an opportunity to show that you're a stand-up company/organization/person. You stand to win greater loyalty and affection by handling the problem effectively. The people who are complaining (assuming they are correct, of course) are helping you by showing you where you're wrong, or at least showing you a different perspective. Squoot "signing" their apology is actually good, in this case, because it indicates the founders (I presume that's who they are) are taking responsibility. It's too bad they flubbed the middle bit.