SAB, DDI and SDMX

In the latest SAB newsletter (October 2011) Arofan Gregory has an item about the “business case” for the use of both DDI and SDMX.

Do you think that his text contains a business case?

Do you think that it contains a solution to the DDI/SDMX integration issue?

Of course not. That would be doing serious work. Attempts in that direction would also reveal that Mr Gregory does not have a clue about what he is doing.

Instead, the only thing we learn from the article is that NSOs may need both DDI and SDMX, because they have been developed for different and only partly overlapping purposes. To cap we are informed that there is such a thing as the DDI/SDMX dialogue (but not whay they have (not) achieved).

This is the wagging-the-dog syndromed all over again. There is DDI, which grew slowly and obliquely into the NSO metadata sphere. Then there is SDMX, which was a one off consultant-cowboy initiative that nobody really understood.

So, you take parts from two different bodies and then try to put them together in a new body. Good luck with that! In fact, to continue the metaphor, they do not even know what a real body looks like nor what parts of such a body that are needed and what parts DDI and SDMX cover.

Here is a thought! What if the solution really is one model, which neither is like DDI nor SDMX. Then what? (I know this to be the case, because I know what the solution is).

Then there are organisational issues, discussed by Martin Vocsan in the editorial.

Sure, the organizational structures need focusing. There is currently more organizational bodies “working” with these issues than there are high level objects in the GSIM model. (Here is the SAB page that shows an overview of the current organizationl structure). However, such a focus is hardly possible before there is a focus for content issues. What exactly are all these groups trying to do, and in what order?

The simple truth is that they just do not know.

The DDI-Alliance and the identity problem

Remember the message from this blog? That core modelling issues remain unsolved?

We have of course again and again heard about combining DDI and SDMX. There is a lot of talk, but no results.

Here is a quote from the DDI-Alliance, from May this year (2011):

Update on DDI Lifecycle 3.2
This update has been delayed because of a complex identification issue. There are also a variety of bugs that will be addressed in this version and the high-level documentation is being revamped as well. This version is likely to be released next winter.

Approach to Identification in DDI Lifecycle
This is a complex issue and currently there is no agreement about the best way to handle it. Consequently, a procedure has been established to solicit outside input in order to arrive at a solution that will meet everyone’s needs. Joachim Wackerow is writing up a summary of the issue and the two proposed approaches. This will be sent to the EC and to a set of experts who will be asked to comment. It is anticipated that this will be followed by a step to synthesize the comments and a Web meeting in which a final proposal is hammered out.

The identification issue is specific to the DDI Lifecycle branch of the specification and has to do with referencing metadata items. IDs should never change and we need to determine what information should be them, what is just related, etc.”

Sorry, these external “experts” will in all likelihood fail to solve this problem! Why? Because the problem has not even been properly framed yet.

Problems integrating models?

Some interesting papers are coming from the recent SDMX expert group meeting.

One is from a consultant that is trying to integrate SDMX with DDI.

This is interesting, since what this consultant is trying to do – without probably knowing it – is to solve a very old NSO modelling problem:

Finding a model that integrates micro and macro data.

From the paper one can guess that this is not so easy. There are incompatibilites. Go figure!

If you have key insights, the solution is simple. I know the solution. Therefore I also know that the solution will not come from mixing DDI with SDMX.

This will be fun to follow!

The METIS steering group strikes back, no. 1

How has the METIS steering group responded to the three messages from this blog?

To recap the messages:

1. You have been going in circles for thirty years

2. You do not have a clue what you are doing

3. The problems you are trying to solve have already been solved

One response is to look back at the “progress” made during the last twenty years – at the next regular UNECE/METIS meeting, in March 2010.

Let us take this task seriously! What progress has the international metadata circuit made?

As anyone who has experience of the use (and abuse) of statistics knows, progress is relative. If you have a 100 yards to travel, and you have covered 4 yards, you have made progress, but you are not even close to your goal.

As we all know, the international metadata circuit is not even close to its goal: truly standard models for alla sub-domains of statistical metadata.

But, surely there has been some progress?! Yes, some, but take a closer look at it!

1. DDI

2. SDMX

3. Karlis Zeila´s (Latvia and Cyprus) metadata-driven production system

Where on this list can we find achievements by the NSO frequent flyers?

1. DDI emerged obliquely, from universities and national data archives, not from the regular self-styled NSO “metadata experts”

2. SDMX was developed more or less single-handedly, by an experienced private consultant. NSO experts could initially not – and most probably still cannot – understand exactly what SDMX is.

3. Karlis Zeila is a uniquely talented individual, who did most of his work on his own, before he emerged on the international statistical metadata scene.

But, what about the technical achievements?

1. DDI has a batch-like, “code book” or “document-oriented” XML structure. This means that it has taken a side track, avoiding the hard core issues of metadata modelling. In fact, it does not even attempt to address those issues!

2. SDMX is also an XML-based standard. Its focus is exchange, not hard core models for integrated NSO systems. The first version was limited to “structures for time series”. This places it in the well-trodden area of macro data models that resemble industry-standards.

3. The Latvian system is an exceptional achievement, but it focuses on solving practical issues, at the cost of models that lack true generality. Surely a powerfull and usefull system, but it does not solve basic metadata modelling issues in a standardised way.

It gets even better! The session about progress the last twenty years will be led by Dan Gillman, from the U.S. Bureau of labor statistics. The number of successfull metadata projects in his office, seems to be inversely correlated to his own influence in the international metadata circuit!

This is logical, since the real raison d´être of this circuit is international travel, back-slapping and buffée dinners, not to solve the hard core statistical metadata modelling issues.