The purpose of the meeting was to discuss and make changes to the draft charter text, and to discuss the next steps. A number of additonal objectives were discussed, including studying the effects of concurrent development and evolution of FDTs and the standards using them, investigating the generic types of tooling that are needed, and looking at the relationship between document quality and the development processes used. It was agreed that participation in the group should be expanded, to engage with the academic community that does not traditionally interact with the IETF.
Notes/Minutes
Stephen opened the meeting and set out the agenda.
Attendees introduced themselves.
Stephen read through the draft charter text, including the background, objectives, and organisation.
Charter text bashing: evolution and tooling
Everyone was asked to bring up the editable document containing the charter text, for discussion and changes.
Carsten added an objective for studying the effects of concurrent development/evolution of FDT techniques and the standards using them.
Carsten: Something that people have been fighting with for some time. Have some techniques like ABNF that are exceedingly stable, there have been changes, and these changes have been adopted very slowly. We need to get a better handle on ways to deal with this concurrent evolution and use of FDTs.
Colin: Very noticeable how little uptake there has been of changes in ABNF. and how long it takes for anything to work its way through to be used.
Carsten: The YANG people did a new version and actually changed their generic data model underneath too, and they do manage to push through these changes. I think it depends on whether this is actively pursued or not.
Colin: The YANG people seem to be progressing quickly. For ABNF, nobody seems to be aware that it has changed in the last 40 years. It hasn't changed much, but it has evolved a little bit.
Marc: I read an RFC yesterday, can't remember which one, where before starting to use ABNF, they made a modification to it that was local to that RFC. I was wondering why this modification, which seems useful, did not make it into the official version of ABNF, as a proper RFC standard.
Ruediger: What is the official version of ABNF? Probably in IETF, one would refer to an RFC, and that obviously doesn't change just because someone has their own private extension. There is actually some process I guess.
Marc: Yes, absolutely. So the official version is 5234. What I mean is why did these people not try to reach consensus to add this to the official version, and publish a new document. So in response to Carsten, it would be interesting to understand why these things are modified locally. I think the best example of that is TLS. There is not one version of the TLS language; each version of TLS redefines the language.
Colin: QUIC adopted something similar but not the same, to make it even harder.
Dave: I think one of the problems with ABNF is the lack of tooling. I was trying to use it to actually validate data, and could not find tools that would allow me to do that. I wound up looking at parser tools instead. Maybe they're not visible, but that was an obstacle.
Stephen: I guess the challenge is to not discourage evolution. YANG example is interesting. YANG vs ABNF, at different points in their lifecycle. Once YANG has matured, will we see the same reluctance to evolve? The challenge for this group is to think about the ways in which evolution can be enabled, even with more mature techniques.
Carsten: When I needed an ABNF tool, I just wrote one.
Ruediger: Not everybody can do this. But I seem to remember there used to be something for at least checking ABNF syntax for RFCs and I-Ds. I think the IETF tools community actually had something. And yes, one thing, one would reasonably do when trying to enhance the language, would be trying to actually help extend those existing tools. Dave, did you look into the IETF toolbox?
Dave: Yes. I did see the validation tools, I didn't see the data processing tools. Wound up trying to make something in a parser called LARC, for Python, and trying to make that look like ABNF, but it wasn't an exact correspondence.
Ruediger: For one point in the IETF protocol spectrum, using BNF style stuff, languages actually defined in the I-D and RFCs, essentially using BNF syntax that could be used as input into the Bison unix parser and scanner tools.
Dave: Lex and Yacc
Ruediger: That has been used, and I like this approach, where the definitions can be easily moved over to the implementation.
Colin: Carsten, you've got the markdown tools - do they integrate parsers for ABNF?
Carsten: No, I was trying to keep that separate. So that is actually an interesting question. So, maybe it's interesting to look at CDDL right now because that's where active tool development is going on. Actually we have one draft that integrates ABNF into CDDL, so we will see some support for ABNF in the CDDL tools, and we have found that it is really useful to have three things in such a tool: validating an FDT-based specification (so you'd validate your ABNF), validating instances against the FDT-based specification (so you'd validate some data/text against the ABNF), and generating random instances. For ABNF, we have bap for validating ABNF specs, and we have something called ABNFgen that generates random instances, but I'm not aware of a tool that gives you validation of instances against the spec out-of-the-box. So that's why I wrote one when I needed it.
Dave: For the third bullet, it was also be nice to generate instances that don't conform. I've found that positive and negative test vectors are both useful.
Marc: Like fuzzing?
Carsten: Yes. Before you fuzz your software, you really want to fuzz your specification, because if you have a random instance generator, you will notice that your specification allows things that you didn't want to allow.
Colin: There's a number of interesting questions about how to integrate this: where it fits into the tooling, and how to fit it into the workflow nicely.
Carsten: The YANG people have some very nice workflows. We could analyse them and derive some more general priniciples for what other FDTs should be using.
Stephen: Thinking about how they fit into the Datatracker tooling. Want to think about the combinations of tooling, and ensure that each validator only triggers for documents that the FDT that they target.
Carsten: Right now in the RFC XML file we can identify, for a piece of source code, what language/syntax that it uses. So we can say something is ABNF, and so on, but we cannot say "this is supposed to be a piece of text that conforms to that ABNF", so that is something we would need to add to the document generation toolchain.
Colin: It's also something which you can put into the text surrounding that, with a few conventions about how you phrase things.
Marc: It would be interesting to add a set of languages that says that this is an instance of a language, and not just a description.
Carsten: Before you can do that, you actually need a way to reference specific pieces of FDT. YANG people have module names, but nothing comparable on the ABNF side.
Ruediger: In IETF speak, we obviously would need some kind of registry (at least one).
Carsten: Maybe an RFC number is good enough for most cases.
Ruediger: You don't want to wait for the 5 years minimum to go from an I-D to an RFC.
Carsten: Yes, so you'd use an I-D name. We don't have to design that right now, I just wanted to highlight that there are other ways to do this.
Colin: I agree that it's worth thinking about the co-development of the techniques and the standards using them, and what we need for the tooling. That seems to be in scope for this group, though we have to be careful about how we interact with the tools team, and avoid seeming like we're taking over tool development.
Ruediger: What we probably cannot do is tell the tools team "please do this". Providing additional code, if we can do code, and offer it to be integrated and checked against the environment, I don't think that is going to be really a problem.
Colin: I don't think it's going to be a problem. I'm cautious of the experience of some of the review teams and research groups, which have perhaps been overzealous about going in and evangelising their approaches in the past. We have to be careful about how we approach things.
Carsten: It's really useful to have running code.
Ruediger: Exactly. Actually, that's something that is missing more than it should be missing in the regular IETF.
Stephen: I think approaching the tools team and trying to understand the current process. For example, how did the ABNF/YANG validators get integrated. So understanding the general process, and just making that easier to access.
Charter text bashing: process and effect on quality
Marc: I'd like to talk about another part of the charter, which is the first sentence of the background. I'm channelling Stéphane Bortzmeyer here, and I agree with him. The first part of the sentence doesn't imply the second part. So "incremental, distributed, and consensus-driven standards development process": there is no proof that this leads to standards documents that contain inconsistencies and ambiguities. I would like to go even further: I think that this should be the subject of a survey or a study to see if the way that we design our standards does lead to inconsistencies/ambiguities, or if these also crop up in outputs from other SDOs, using different techniques.
Stephen: So there are a couple of things here. I think we can rewrite these couple of sentences to make it clear. What I meant here is that the incremental/distributed/consensus-driven process implies the need for natural language, and that it is this that leads to inconsistencies and ambiguities. I agree that as written it doesn't follow.
Marc: I don't understand why incremental/distributed/consensus-driven development implies natural language.
Ruediger: Well, if you move to anything different, you are excluding a lot of the current participants.
Marc: I'm not saying that we should do something else, I'm just saying that I'm not sure that the current process mandates using natural language.
Ruediger: If you want to avoid the natural language, you have kind of back-pressure that implies that the community that does the consensus-driven stuff gets very small.
Marc: Let me find another way to explain this. When we develop software, so say the Linux kernel, it's using a formal language, but it is using an incremental/distributed/consensus-driven development process. So it is not a consequence of having an incremental approach that we use natural language. We use natural language for other reasons. I'm not saying this just to argue about the sentence, but I would be really interested to see if the inconsistencies/ambiguities arise because of the way in which we develop standards, or if it is a natural thing that would happen anyway.
Colin: Yes, I think that's an interesting question. Is it a fundamental consequence of doing things in this very open way, or is it a consequence of the details of how we develop the standards?
Ruediger: I tend to say that the consequence is essentially based on humans being humans, and being involved.
Marc: This would be my assumption too, but the sentence was not saying this. To make things clear: for example, does the ITU produce standards that are better than the IETF standards? We use different ways of doing things. Does the process used change the quality of the end documents? I think this is not just for the first sentence, I think we should study this. It would be interesting to have a definitive answer to that.
Colin: Yes, and I think that's something where we may actually be able to come up with a definitive answer. Not in terms of does this result in better protocols, but rather, does it result in fewer errors of a certain type in the specifications. "Better" is obviously relative to what your goals are, but "does it contain inconsistencies" is something that perhaps can be evaluated empirically.
Stephen: Is that work that you've already started, Marc, in terms of going through the errata, and labelling/tagging them?
Marc: Yes. I think that if you could put in a database all the mistakes that we've made -- and this is a huge task -- we are close to 9000 RFCs. To label each errata and each RFC to say, "in this RFC we use this formal language", "we use this technique", and to then compare this to the rate of error. Errata are a relatively good measurement. Unfortunately errata have been kind of neglected. It's not easy but this could be, in my opinion, a good way to measure exactly what works, what doesn't work, and perhaps give us a path to where we should go.
Colin: We can also look at updates to specifications. If it's the ITU, different versions of the recommendations at different years, if it's the IETF, then updates to previous RFCs and so on, to see if there are bugs fixed, rather than just features enhanced.
Marc: I think that everyone's favourite example is DNS. There was one specification written a long time ago, and there have been more than 30 drafts that modify it. I'm not talking about new record types, but modifications to the protocol. So there is probably something to say about this.
Colin: But also just look at SIP or RTP or STUN, or anything on the multimedia side. None of them are particularly fixed in stone on the first draft. Or TCP, which keeps getting extended.
Marc: So, I don't know if we have an objective on this. Perhaps someone can wordsmith an objective to study that.
Stephen: So I've tweaked the first couple of sentences in the background, I don't know if that's better, and I've added an objective to try to study the impact of FDTs on the quality of standards documents, where quality is inconsistencies/ambiguities/amount of errata that get produced. Does that make sense?
Marc: Yes.
Colin: I think it makes sense. I think we've got a lot of objectives. If we're reading this as a "this group is going to" rather than a "this group could", then I think we'll need more than seven people to do these objectives.
Stephen: I think some of these, especially the newer ones, fall within the more general objectives that we already have. So, we're essentially looking at the cultural and the social aspects, so that covers a couple of the new ones, and then the technical side aswell. I think the general ones overlap what we're proposing here.
Colin: So we might state the general objectives, and then say, "for example..".
Stephen: Yes. I think these three at the bottom of the list fall into the more broad ones, so I can collapse them in as examples. Is there anything else about the text that we should discuss?
Marc: It's fine.
Stephen: Will make the proposed changes, and circulate with the meeting materials.
Dave: Under the set of generic tools, we had also discussed how to include examples in a specification, and validate those examples against a particular FDT specification. Not just the tool for doing validation, but also for extracting instances to be validated.
Marc: Think this is covered by the second bullet point that Carsten added.
Dave: OK, yes. If it can parse the entire spec, then validate the instances, that's covered.
Ruediger: From where I'm coming, I think I am missing one point of view, that is, dealing with presentation layer. In one direction, the specification of how stuff that might be specified in ABNF down to presentation in the bits, or the usual ASCII art of PDUs, getting that specification done in reasonable, simple, formal language. The other direction, when you look in to some areas of the protocol universe, you find that people are using TLV structures (type-length-value), but when you start to build data structures for PDUs, with multi-level TLVs, the specification done in ASCII art, plus natural language, gets very much out-of-hand, and when you see a WG presentation on something like that, you might think that people are essentially, prohibited from seeing the contextual stuff that could probably described in typical programming language. Quality of the protocols and the implementations could be much improved by using more formal languages in those areas. The presentation layer requirements are probably quite different in the different protocol domains.
Stephen: I think that makes sense; falls under the "technical barriers" objective. May be worth adding a specific example.
Colin: These look like good examples that fit within that broad theme.
Stephen: Agreed. Developing a taxonomy of different types of specification and related FDTs is a good idea. Fits within a couple of the objectives. Is there anything else on the text?
Colin: Just need someone to take these ideas and fold them into the text and tidy it up.
Stephen: Will do that, and circulate it with the meeting materials.
Discussion: Engagement and interaction
Stephen: Need to think about how we start meeting the objectives, in terms of engagement, both with the standards community, and the academic research community. Need to think about who we want to talk to, and how.
Colin: Think we've discussed a fair chunk of this already.
Stephen: Perhaps in terms of how we engage, but are there any people or groups that we should think about targeting for future meetings?
Marc: The community of people that are using FDTs is spread across WGs, so we don't have particular groups; that's why we've created this group.
Colin: Things like the Hackathon are quite good for giving visibility. HotRFC talks, and things like that. It's mostly about spotting people doing relevant things and going and having a chat with them, sharing experience, seeing how things build up from there. More interesting is how we broaden the community, particularly bringing in the research community, which might have ideas that we can use, but that don't currently talk to the IETF.
Marc: Yes, I think that this is, from my point of view, the goal that I have most interest in. There is a large amount of history in CS of working on these things. It would be fantastic to have some of these people in the group.
Stephen: Guess the challenge in terms of doing that is identifying the right people/venues. I don't have that experience; not sure if anyone knows people working in that area.
Colin: So, obviously there's a couple of people in Glasgow that do protocol validation, and we should clearly talk to them. I know SIGCOMM has had the NetPL workshop running a view times. There's the routing algebra stuff aswell; people like that that we could reach out to relatively easily. I'm sure I've forgotten a bunch of them.
Stephen: Next step to organise a more technical meeting and invite some of these people that aren't here to come along.
Colin: MODELS conference looks like a good venue.
Marc: I think in Scotland there is a large community working on stuff related to what we are doing. I mean, ABCD session types work is directly related.
Colin: Session types stuff, at least partly, comes out of Glasgow so we can talk to those people very easily. Have talked to some of them on occasion. Obviously, you're interested in Idris coming out of Scotland too. Not sure what we have in the rest of the community. The SIGCOMM stuff, and people like Nate Foster (Cornell), so there's a community in the US. Routing algebra folks; Tim Griffin. Not sure what there is in Europe. Anyone in Germany?
Carsten: People looking at things like XML.
Marc: Spent a few years reading about this stuff. Hotspot currently is Scotland, without doubt.
Colin: Sure. We're certainly talking to those people. Maybe the thing to do is just to try and kickstart a workshop, and get some of the people from that community involved, expand things that way.
Discussion: Future meetings/next steps
Stephen: Next meeting should be more technical, showcase of work, bring in people that aren't part of the IETF community. End of next month?
Colin: What's the goal?
Stephen: To broaden the set of people coming along - show case each other's work/ideas. Would that make sense?
Colin: It makes sense, but need to reach out people, and think about what the next meeting should be about. Coordinating a workshop, or a meeting about technical work?
Carsten: One way to get the interest of some people. ASDF chartering process, where Rob Wilton suddenly said, what about YANG, how does it fit with YANG. Document that outlines how to translate. Many people that are working on some techniques are interested in looking at related techniques, and thinking about how they work together and replace each other and so on.
Ruediger: Looking at operational use of protocols/specifications, one part of my idea on presentation layer references is that one of the goals, I think, in better protocol specifications would be to help to create the relation between the protocol definition, and the related YANG models within the specification. Instead of having the protocol and the YANG model separately. Getting all of these things covered right in the beginning can be done and would be extremely helpful.
Stephen: Thinking about the next steps. Will bash on the charter text a bit more. We can each think about how we can reach out to the IETF and academic communities. Next meeting should be about coordinating a workshop-style meeting to bring people together.
Marc: One idea for IETF engagement is tutorials/edu. Maybe people are not aware of what FDT is and why FDT is useful. A couple of tutorials, like have been given on Sundays before the IETF traditionally, could be a good way to recruit people.
Stephen: Interesting idea.
Colin: Start reaching out to more people, and see if there's interest in bringing something together, with a view towards doing a workshop, co-located with IETF or a conference. Just to start bringing the two communities together.
Dave: One of the interesting parts of the SDF charter review was comparing and contrasting SDF to YANG, and the charter came up with a good distinction between the purposes of the two. One thing we might want to do is to look at YANG itself, and compare it to the charter here, and identify the gaps.
Stephen: Next steps - bash on charter text, circulate, each think about people to reach out to. Then we can think about what sort of forum makes sense to bring these people together. Next meeting to discuss logistics of that.
Marc: Would be good to have a list of conferences that are in scope.
Stephen: Not got an answer to that yet, but will compile this as part of reaching out to people in Glasgow.
Colin: Don't just reach out to people in Glasgow, but more widely.