International Federation of Digital Seismograph Networks
Thread: FW: fdsnsws-station and stationXML1.1
Philipp Kästli
Dear colleagues,
First I apologize to WG3 members which are also on the EIDA technical mailing lists for cross-posting - the mail below is an implicit answer to the previous mail of Jean-Marie.
Summarizing:
I am absolutely in favor of quickly adopting stationXML 1.1, and of introducing it in fdsn/station web services.
My only point is that ...
- in order to comply with the versioning scheme defined in the FDSNWS standard,
- in order not to break any fdsnws/station/1/ clients,
- (thus, in order to increase trust in fdsn standards and procedures),
... station metadata in stationXML 1.1 format should be distributed over a fdsn station service v. 2.0, thus under the URL fdsnws/station/2/ .
I suggest that the FDSN as holder of the standard quickly declares this new version and avoids all ambiguity before heterogeneous treatment of the issue starts to creep into the worldwide population of fdsn station services.
Kind regards,
Philipp
********************
Dear Jean-Marie,
With https://www.xml.com/pub/a/2003/12/03/versioning.html (1), you actually point to an excellent read.
However before discussing its implication, I have to clarify on the term *compatibility*, as the document (1) uses it in a different way.
(1) describes a SOAP context, where a customer sends an XML instruction to a service, which in return does something. Here, a backwards compatible service means that the new service version (implemented for digesting the new version of the instruction xml) is still capable to interpret the old one. Technically it means that the old xml fulfills also the new schema. Usability-wise (and I think this is the key point) it means that the new service version does not break the user expectation of a customer who is not aware of the service version change.
(1) fdsnws (https://www.fdsn.org/webservices/FDSN-WS-Specification-Commonalities-1.2.pdf) describes a REST context, where a customer sends a GET request and, in return, receives a stationXML. A backwards compatible service, from a customer's perspective, means that a change of the service fulfills both requirements a) the service can understand an unchanged get request (a get request of a customer which is not aware of the new minor version, or implementation of the service), and b) the service returns a result a client unaware of the new service minor version or implementation "can understand". "can understand", in the context of a standard defined in xsd, means: it fulfills (validates against) the unchanged (old) schema.
Only if a) and b) are met, a customer can go on using the updated service without being aware of the update. (thus without the need from the service provider's side to announce the change, and give the customer time for adaptation.
Now I admit that this definition of compatibility relies more on common sense than on a literature source. However it seems in line with the intention of the authors of the FDSNws standard: they requested that a not compatible = major service upgrade should be made transparent to the customer by a change of the URL (more precisely, the major version number in the URL). A changed URL is a very powerful way to tell the user of the old URL that things are not as they were.
Using this definition, an fdsn/station service returning StationXML 1.1 documents must have a different major version than its precedecessor returning StationXML 1.0, *if* a StationXML 1.1 *document* does *not* validate against the StationXML 1.0 *schema* definition (the one used by the update-unaware client to validate the response). Not vice versa.
Now, stationXML 1.0 and 1.1 have the same namespace, which makes the same schemata at least applicable, and the changing version number does not per se break validation. So the question why a stationXML 1.1 document is not, at the same time, valid stationXML 1.0 is a justified one.
It is not. XML works not in a way that you can, by default, just add anything anywhere, and if it is unknown to the schema, it is ignored. But you are required to define where "something unknown" can be added, and you can restrict what "unknown" means. This has been done in stationXML 1.0 - look in the schema for <xs:any> for any element, and xs:anyattribute for any attribute. However, you will see that all these extension points have the namespace="##other" attribute, thus these random add-ons to stationXML documents can be *anything but* elemnts defined in the stationXML scheme (why? To avoid ambiguity**; see below). As the new elements added in stationXML 1.1 are (obviously) part of stationXML, they do not qualify as "any unknown add-on" as allowed according to the 1.0 scheme.
So all this is no argument against stationXML 1.1, or fdsn station web services. It is just an argument to offer stationXML 1.1 only on an fdsnws/station/2/query? service url, in order to make sure the customer is aware of the fact that he gets something he/she has to treat technically differently from what he gets from an fdsnws/station/1/query? service URL.
I posted this issue on the FDSN WG 3 mailing group first as they are holders of both the stationXML and the fdsnws specifications, and should provide a fdsnws/station/2/ specification document with clarification on the topic. Although the practical implications are probably largest in Europe (aka EIDA), where scientists have to deal with many different service endpoints, potentially updated at a different pace...
Best regards,
Philipp
***********
** on ambiguity: imagine you have an staxml:station element type with a sequence of staxml:latitude (minOccurs: 1, maxOccurs: 1), a staxml:longitude (minOccurs:1, maxOccurs: 1), and xs:any (minOccurs:0, maxOccurs: indefinite) child elements. Now, if you allow the xs:any to be *really* anything (including elements from the staxml scheme), a user could do this:
<station code="BLA">
<latitude>47.0</latitude>
<longitude> 7.0</longitude>
<latitude>48.0</latitude>
</station>
, which is, at the same time valid and not valid, and it is definitely not what you would like to have passed to a staxml-aware software code...
-----Original Message-----
From: eida_maint<at>gfz-potsdam.de [eida_maint<at>gfz-potsdam.de]
Sent: Donnerstag, 5. Dezember 2019 11:52
To: eida_maint<at>gfz-potsdam.de
Subject: Re: fdsnsws-station and stationXML1.1
Dear Philipp,
Thanks for those clarifications.
I'm also working on stationXML outside RESIF, I let them answer from their perspective.
And since I will have, among other things, polynomial responses, I must use 1.1 version.
* Anyway, you might have understood I'm far from an XML expert.
I previously understood that any tag addition in an XML file (I'm not speaking specifically about stationXML here) was not a problem as clients can be made to only read a subset of them and ignore any unknown tag.
Which was perfectly fine for me.
And from your answer, I'm puzzled because I understand that any tag addition will break the backward compatibility.
This would mean we would never be able to do any revision of an XML standard without breaking the compatibility.
* Could it be because you are trying to validate a 1.1 document with regards to a 1.0 standard ?
This obviously will break because, as it is written in the stationXML header, it's a 1.1 version, not a 1.0 version.
* Quickly browsing through the web, the XML standard has an "Ignore Rule" that says precisely that a client "Must ignore" an unknown tag.
https://www.xml.com/pub/a/2003/12/03/versioning.html
https://www.goland.org/xmlschemabackwardscompat/
Is that wrong ?
Regards.
Jean-Marie.
Le 05/12/2019 à 09:35, Kästli Philipp kaestli<at>sed.ethz.ch a écrit :
--------------------------------------
Institut de Physique du Globe de Paris
Observatoires Volcanologiques et Sismologiques
+33 1 83 95 74 37
+33 6 20 35 28 04 (portable)
1 rue Jussieu
75238 Paris cédex 5
#############################################################
This message is sent to you because you are subscribed to
the mailing list <eida_maint<at>gfz-potsdam.de>.
To unsubscribe, E-mail to: <eida_maint-off<at>gfz-potsdam.de> To switch to the DIGEST mode, E-mail to <eida_maint-digest<at>gfz-potsdam.de>
To switch to the INDEX mode, E-mail to <eida_maint-index<at>gfz-potsdam.de> Send administrative queries to <eida_maint-request<at>gfz-potsdam.de>
First I apologize to WG3 members which are also on the EIDA technical mailing lists for cross-posting - the mail below is an implicit answer to the previous mail of Jean-Marie.
Summarizing:
I am absolutely in favor of quickly adopting stationXML 1.1, and of introducing it in fdsn/station web services.
My only point is that ...
- in order to comply with the versioning scheme defined in the FDSNWS standard,
- in order not to break any fdsnws/station/1/ clients,
- (thus, in order to increase trust in fdsn standards and procedures),
... station metadata in stationXML 1.1 format should be distributed over a fdsn station service v. 2.0, thus under the URL fdsnws/station/2/ .
I suggest that the FDSN as holder of the standard quickly declares this new version and avoids all ambiguity before heterogeneous treatment of the issue starts to creep into the worldwide population of fdsn station services.
Kind regards,
Philipp
********************
Dear Jean-Marie,
With https://www.xml.com/pub/a/2003/12/03/versioning.html (1), you actually point to an excellent read.
However before discussing its implication, I have to clarify on the term *compatibility*, as the document (1) uses it in a different way.
(1) describes a SOAP context, where a customer sends an XML instruction to a service, which in return does something. Here, a backwards compatible service means that the new service version (implemented for digesting the new version of the instruction xml) is still capable to interpret the old one. Technically it means that the old xml fulfills also the new schema. Usability-wise (and I think this is the key point) it means that the new service version does not break the user expectation of a customer who is not aware of the service version change.
(1) fdsnws (https://www.fdsn.org/webservices/FDSN-WS-Specification-Commonalities-1.2.pdf) describes a REST context, where a customer sends a GET request and, in return, receives a stationXML. A backwards compatible service, from a customer's perspective, means that a change of the service fulfills both requirements a) the service can understand an unchanged get request (a get request of a customer which is not aware of the new minor version, or implementation of the service), and b) the service returns a result a client unaware of the new service minor version or implementation "can understand". "can understand", in the context of a standard defined in xsd, means: it fulfills (validates against) the unchanged (old) schema.
Only if a) and b) are met, a customer can go on using the updated service without being aware of the update. (thus without the need from the service provider's side to announce the change, and give the customer time for adaptation.
Now I admit that this definition of compatibility relies more on common sense than on a literature source. However it seems in line with the intention of the authors of the FDSNws standard: they requested that a not compatible = major service upgrade should be made transparent to the customer by a change of the URL (more precisely, the major version number in the URL). A changed URL is a very powerful way to tell the user of the old URL that things are not as they were.
Using this definition, an fdsn/station service returning StationXML 1.1 documents must have a different major version than its precedecessor returning StationXML 1.0, *if* a StationXML 1.1 *document* does *not* validate against the StationXML 1.0 *schema* definition (the one used by the update-unaware client to validate the response). Not vice versa.
Now, stationXML 1.0 and 1.1 have the same namespace, which makes the same schemata at least applicable, and the changing version number does not per se break validation. So the question why a stationXML 1.1 document is not, at the same time, valid stationXML 1.0 is a justified one.
It is not. XML works not in a way that you can, by default, just add anything anywhere, and if it is unknown to the schema, it is ignored. But you are required to define where "something unknown" can be added, and you can restrict what "unknown" means. This has been done in stationXML 1.0 - look in the schema for <xs:any> for any element, and xs:anyattribute for any attribute. However, you will see that all these extension points have the namespace="##other" attribute, thus these random add-ons to stationXML documents can be *anything but* elemnts defined in the stationXML scheme (why? To avoid ambiguity**; see below). As the new elements added in stationXML 1.1 are (obviously) part of stationXML, they do not qualify as "any unknown add-on" as allowed according to the 1.0 scheme.
So all this is no argument against stationXML 1.1, or fdsn station web services. It is just an argument to offer stationXML 1.1 only on an fdsnws/station/2/query? service url, in order to make sure the customer is aware of the fact that he gets something he/she has to treat technically differently from what he gets from an fdsnws/station/1/query? service URL.
I posted this issue on the FDSN WG 3 mailing group first as they are holders of both the stationXML and the fdsnws specifications, and should provide a fdsnws/station/2/ specification document with clarification on the topic. Although the practical implications are probably largest in Europe (aka EIDA), where scientists have to deal with many different service endpoints, potentially updated at a different pace...
Best regards,
Philipp
***********
** on ambiguity: imagine you have an staxml:station element type with a sequence of staxml:latitude (minOccurs: 1, maxOccurs: 1), a staxml:longitude (minOccurs:1, maxOccurs: 1), and xs:any (minOccurs:0, maxOccurs: indefinite) child elements. Now, if you allow the xs:any to be *really* anything (including elements from the staxml scheme), a user could do this:
<station code="BLA">
<latitude>47.0</latitude>
<longitude> 7.0</longitude>
<latitude>48.0</latitude>
</station>
, which is, at the same time valid and not valid, and it is definitely not what you would like to have passed to a staxml-aware software code...
-----Original Message-----
From: eida_maint<at>gfz-potsdam.de [eida_maint<at>gfz-potsdam.de]
Sent: Donnerstag, 5. Dezember 2019 11:52
To: eida_maint<at>gfz-potsdam.de
Subject: Re: fdsnsws-station and stationXML1.1
Dear Philipp,
Thanks for those clarifications.
I'm also working on stationXML outside RESIF, I let them answer from their perspective.
And since I will have, among other things, polynomial responses, I must use 1.1 version.
* Anyway, you might have understood I'm far from an XML expert.
I previously understood that any tag addition in an XML file (I'm not speaking specifically about stationXML here) was not a problem as clients can be made to only read a subset of them and ignore any unknown tag.
Which was perfectly fine for me.
And from your answer, I'm puzzled because I understand that any tag addition will break the backward compatibility.
This would mean we would never be able to do any revision of an XML standard without breaking the compatibility.
* Could it be because you are trying to validate a 1.1 document with regards to a 1.0 standard ?
This obviously will break because, as it is written in the stationXML header, it's a 1.1 version, not a 1.0 version.
* Quickly browsing through the web, the XML standard has an "Ignore Rule" that says precisely that a client "Must ignore" an unknown tag.
https://www.xml.com/pub/a/2003/12/03/versioning.html
https://www.goland.org/xmlschemabackwardscompat/
Is that wrong ?
Regards.
Jean-Marie.
Le 05/12/2019 à 09:35, Kästli Philipp kaestli<at>sed.ethz.ch a écrit :
Dear Jean-Marie,--
that stationXML 1.1 breaks compatibility with stationXML 1.1 you can
verify easily by actually validating an 1.1 document against the
stationXML 1.0 schem:
https://www.fdsn.org/xml/station/fdsn-station-1.0.xsd
nonix:~/eida/stationxml$ curl
'https://www.fdsn.org/xml/station/fdsn-station-1.0.xsd' >
fdsn-station-1.0.xsd nonix:~/eida/stationxml$ curl
'https://ws.resif.fr/fdsnws/station/1/query?network=G&station=ECH&leve
l=station&format=xml'>
resif.xml
xmllint --schema fdsn-station-1.0.xsd resif.xml <?xml version="1.0"
encoding="UTF-8"?> <FDSNStationXML
xmlns="http://www.fdsn.org/xml/station/1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:resif="http://www.fdsn.org/xml/station/1/resif"
schemaVersion="1.1"
xsi:schemaLocation="http://www.fdsn.org/xml/station/1
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd">
[...]
</FDSNStationXML>
resif.xml:7: element Identifier: Schemas validity error : Element
'{http://www.fdsn.org/xml/station/1}Identifier': This element is not
expected. Expected is one of (
{http://www.fdsn.org/xml/station/1}Comment,
##other{http://www.fdsn.org/xml/station/1}*,
{http://www.fdsn.org/xml/station/1}TotalNumberStations,
{http://www.fdsn.org/xml/station/1}SelectedNumberStations,
{http://www.fdsn.org/xml/station/1}Station ).
resif.xml fails to validate
nonix:~/eida/stationxml$
XML as a format family is generally chosen because it is defined which
tags must, must can be present (and which tag contents are valid), and
bhus, because an xml document can easily be verified for validity
against the respective schema.In an xml-based environment, the
possibility to validate is a must.
Even if you try to cheat (e.g. by declaring the 1.1 document as 1.0
and/or skipping validation), any client software generated against the
stationXML 1.0 scheme will fail to handle it, e.g. as the XML document
makes use of the stationXML 1.1 option that any BaseNode type (station
and channel) can have a persistent identifier (which was given in
stationXML 1.0), or if it lacks a creationDate (which is allowed to
lack in 1.1, but forbidden to lack in 1.0), if it has an operator at
the network level (allowed in 1.1, not allowed in 1.0), if it has a
subject or lacks an id in a comment (both not allowed in 1.0), or similar cases.
For the full changes list, see
https://www.fdsn.org/xml/station/fdsn-station-changes-1.0-to-1.1.txt
Kind regards, Philipp
----------------------------------------------------------------------
--
*From:* eida_maint<at>gfz-potsdam.de <eida_maint<at>gfz-potsdam.de> on
behalf of Jean-Marie SAUREL saurel<at>ipgp.fr <eida_maint<at>gfz-potsdam.de>
*Sent:* 05 December 2019 09:47
*To:* eida_maint<at>gfz-potsdam.de
*Subject:* Re: fdsnsws-station and stationXML1.1 Good morning
everyone,
I also don't clearly understand why the stationXML 1.1 breaks
compatibility with stationXML 1.0.
Do you have any more details on that ?
The only item that stationXML 1.1 removed is the StorageFormat tag,
which, as it was pointed in FDSN WG3, is very unlikely to have any use.
Furthermore, this tag is not a mandatory tag according to stationXML
1.0 xsd (minoccurs=0).
The other main modification is the correction of an error in
stationXML
1.0 polynomial response representation.
The 1.0 version request a StageGain to be present event in a
polynomial response, which simply makes no sense and is against SEED manual 2.4.
The only way I imagine a 1.1 stationXML could break a client coded for
1.0 would be because of this missing StageGain in polynomial response.
Otherwise, a correctly coded stationXML 1.0 should not break if there
are optional or non-mandatory missing tags.
Regards.
Jean-Marie.
Le 05/12/2019 à 08:30, Jonathan Schaeffer
jonathan.schaeffer<at>univ-grenoble-alpes.fr a écrit :
Hi John and all,--
I'd like to emphasize that RESIF does not break any specification of
fdsnws:station1.0 (as you said).
If we (ETC) are to identify the best options, we need to understand
why
stationXML1.1 "breaks any clients".
We did not any complaint from users using Obspy as far as I know.
The suggestion (having fdsnws:station2.0 and fdsnws:station1.2
specifications and implementations) for this minor change will be a
real pain for everyone, and in particular for the end users. I see a
lot of usage drawbacks and won't elaborate here.
If this issue is an emergency, I suggest we plan a visio-conference
of ETC on this point. If it can wait untill the next ETC meeting,
let's put it on the agenda.
Cheers,
Jonathan
On 04/12/2019 22:07, Clinton John jclinton<at>sed.ethz.ch wrote:
Dear EMB and ETC,
You may recall a new version of stationXML was released by FDSN in
May last year, a minor version increase from 1.0 to 1.1.
A few days ago, most of you will have noticed an email from Philipp
and Daniel to the FDSN WG3. They spotted that the RESIF node updated
their fdsnws-station to provide XML in stationXMl1.1 format.
Currently, we do not find any other fdsnws-event service that
returns stationXML1.1, even outside of EIDA. Inside EIDA, this
inconsistency, and lack of control on the format received from
fdsnws-station, breaks any clients, including the federator, and
will cause downstream issues for other users. For example, now a
simple request for metadata from AlpArray Z3 network will provide information in both formats.
Unfortunately, there was no migration plan for the corresponding
fdsnws-station service - the specification is loose and 'allows'
distribution of different minor versions. As I understand, the
situation with different EIDA nodes providing different versions of
stationXML from the same service is not acceptable in the long term.
A proposed plan that 1) ensures immediate re-compliance across EIDA
and
2) provides a definite path to all migrate to the latest stationXML
version could be:
- RESIF reverts back to releasing stationXML1.0 from
fdsnws-station1.1 as soon as possible.
*/(this unblocks the Federator developments)/*
- EIDA requests to FDSN WG3 that
a minor version fdsnws-station1.2 is released, clearly specifies it
returns stationXML1.0 format a major version fdsnws:station2.0, is
released, clearly specifies it returns stationXML1.1 format
- An engineering phase is required to ensure all nodes can provide
both
fdsnws:station1.2 and 2.0. For nodes that provide fdsnws via SC3,
this requires updating the software, most reasonably by gempa,
followed by installation of the new SC3 version at all nodes.
- all EIDA nodes would then support both
fdsnws:station1.2 to deliver stationXML1.0
fdsnws:station2.0 to deliver stationXML1.1 */(then the Federator can
also support stationXML1.0 or 1.1, depending on configuration of the
routing service)/*
I have discussed the issue with Helle today. She notes RESIF is not
currently in a position to provide feedback or take actions for a
couple of weeks and may not actually be able to revert to stationXML1.0.
Further, she notes this plan is relatively complex and may take
time - which is true. On the other hand, this is an opportunity for
EIDA to insist the fdsnws definitions become tighter and more
precise, and to point out future format changes need to be carefully
coupled with sufficient and appropriate changes to exchange formats.
In the meantime, I would really appreciate opinions from the EIDA nodes.
Would others agree to the proposed plan, or suggest another plan or
even an entirely different strategy? Ideally, we would be able to
quickly come to a consensus, approach WG3 and speak on behalf of
EIDA on this issue If needed, we at ETH are happy to organise a
conference call with interested parties on this issue.
Regards,
John
*From:* Kästli Philipp
*Sent:* Donnerstag, 28. November 2019 16:01
*To:* FDSN Working Group III <fdsn-wg3-products<at>lists.fdsn.org
<fdsn-wg3-products<at>lists.fdsn.org>>
*Subject:* stationXML version 1.1 and fdsnws/station service
version 1.x
Dear colleagues of FDSN WG3,
Our last fdsnws/station specification is 1.2 and dates
(content-wise) of April 2, 2019 (an update in June refers to the
formatting of the standards document only).
According to this document, the fdsnws/station response format is
"StationXML" (or text), without indication of the version, and
thus, implicitly, referring to StationXML 1.0
On May 3, 2019, FDSNwg 3 published a new version 1.1 of StationXML.
John and Chad describe the the compatibility situation as follows:
"Additionally, a number of backwards-compatible changes suggested
at previous meetings and on the project site were incorporated.
Conforming to the versioning rules, all 1.0 version documents are
also compliant with schema version 1.1, with one exception: the
<StorageFormat> element was removed from the schema, but is not
widely used and not expected to cause incompatibility."
We would summarize this as follows:
- stationXML 1.0 remains forward-compatible with 1.0, thus
an "old" xml document is still readable for software implemented
for the
1.1 format (with the exception mentioned)
- stationXML 1-1 is *not* backward-compatible with 1.1,
thus a new document violates the old specification and is not
readable for software implemented against the 1.0 specification.
This has implications on the fdsnws/station/1/ web service. The
standards document v. 1.2 says the following on service versioning:
"Web service implementations are versioned according the following
three-digit (x.y.z) pattern:
SpecMajor . SpecMinor . Implementation Where the fields have the
following meaning:
SpecMajor : The major specification version*, all implementations
sharing this SpecMajor value will be backwards compatible with all
prior releases.* Values start at 1. " And "The following base URI
pattern is to be used at each data center implementing FDSN
services: <site>/fdsnws/<service>/*<majorversion>*/"
Thus, an FDSNWS user can expect that if she/he has implemented an
fdsnws-station client according to the standard (thus, capable of
digesting the stationXML of the time, e.g. 1.0), this client will
never break, as long as she/he addresses the service under the same
url ./fdsnws/station/*1*/. .
An updated service with a non-backward-compatible response should
have a different url ./fdsnws/station/2/. So, at the time being, it
violates the fdsnws/station v. 1.x standard, and the justified
expectation of some users, if a service fdsnws/station/1/ 1.x
service returns the new stationXML 1.1 format.
(at the same time, an fdsnws/station service provider of 2020 may,
when reading the fdsnws/station/1/ specification, may not pay
attention that it pre-dates the stationXML 1.1 version, and
consider returning that very format)
Proposed way out:
- have a new fdsnws/station/1/ specification version
1.2(.1)
with the only change that it explicitly says that it is referring
to stationXML 1.0 as its response format
- have a fdsnws/station/2/ specification 2.0 saying that
the response format is 1.1 or any *backward* compatible upper
version
Kind regards,
Daniel Armbruster
Philipp Kästli
Swiss Seismological Service @ ETHZ
--------------------------------------
Institut de Physique du Globe de Paris Observatoires Volcanologiques
et Sismologiques
+33 1 83 95 74 37
+33 6 20 35 28 04 (portable)
1 rue Jussieu
75238 Paris cédex 5
#############################################################
This message is sent to you because you are subscribed to
the mailing list <eida_maint<at>gfz-potsdam.de>.
To unsubscribe, E-mail to: <eida_maint-off<at>gfz-potsdam.de> To switch
to the DIGEST mode, E-mail to <eida_maint-digest<at>gfz-potsdam.de>
To switch to the INDEX mode, E-mail to
<eida_maint-index<at>gfz-potsdam.de> Send administrative queries to
<eida_maint-request<at>gfz-potsdam.de>
--------------------------------------
Institut de Physique du Globe de Paris
Observatoires Volcanologiques et Sismologiques
+33 1 83 95 74 37
+33 6 20 35 28 04 (portable)
1 rue Jussieu
75238 Paris cédex 5
#############################################################
This message is sent to you because you are subscribed to
the mailing list <eida_maint<at>gfz-potsdam.de>.
To unsubscribe, E-mail to: <eida_maint-off<at>gfz-potsdam.de> To switch to the DIGEST mode, E-mail to <eida_maint-digest<at>gfz-potsdam.de>
To switch to the INDEX mode, E-mail to <eida_maint-index<at>gfz-potsdam.de> Send administrative queries to <eida_maint-request<at>gfz-potsdam.de>
-
Dear Philipp,
I am in agreement with the sentiments of Jean-Marie and Philip C. From the perspective of a reader/parser, StationXML 1.1 can be read as if it were 1.0, with the small exceptions already noted. The issue you raise appears to be limited to validation of new documents against the older schema(s) and not with reading the metadata per-se.
As noted, the version is encoded in the XML instance. Any software that wishes to validate can almost trivially determine and use the appropriate schema. So the worst case scenario at the moment is: systems that wish to validate need to be updated to use the latest major-version schema as extensions are created. Or be a bit more clever and check for a declared namespace of "http://www.fdsn.org/xml/station/1" and trust the schemaLocation if it is included in the instance document.
You raise a validate point, we should be clear on the scale of the problem before considering if and how it should be addressed.
Using this definition, an fdsn/station service returning StationXML 1.1 documents must have a different major version than its precedecessor returning StationXML 1.0, *if* a StationXML 1.1 *document* does *not* validate against the StationXML 1.0 *schema* definition (the one used by the update-unaware client to validate the response). Not vice versa.
It was not the intention to define backwards compatible as "must validate against the oldest schema (for the major version)". That would be nice if possible. Unless we find a different way to encode the extension, such a requirement, as Jean-Marie speculated, would appear to make it impossible to extend StationXML without breaking it's backwards capability. If we extend this logic to the fdsnws-station service, this would mean a new major version for any change to StationXML no matter how small.
Keep in mind that creating a new fdsnws-station major version has significant implications, new service endpoints by each center (probably running in parallel with the old ones for an extended time), updates to a lot of software that is coded with these URLs and more wrinkles. This is not something we should do trivially.
Perhaps there is a clever way to modify the schema (e.g. new elements would have a different namespace), such that extensions can be created while allowing the oldest schema to validate documents with the extensions. I will look into it next year and encourage others to investigate as well. Such a change may have implications for generators of documents, but as 1.1 is not yet widespread and could theoretically be deprecated and replaced with 1.2, this would be a much easier change than making a new service endpoint standard.
regards,
Chad
On Dec 5, 2019, at 10:14 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch> wrote:
Dear colleagues,
First I apologize to WG3 members which are also on the EIDA technical mailing lists for cross-posting - the mail below is an implicit answer to the previous mail of Jean-Marie.
Summarizing:
I am absolutely in favor of quickly adopting stationXML 1.1, and of introducing it in fdsn/station web services.
My only point is that ...
- in order to comply with the versioning scheme defined in the FDSNWS standard,
- in order not to break any fdsnws/station/1/ clients,
- (thus, in order to increase trust in fdsn standards and procedures),
... station metadata in stationXML 1.1 format should be distributed over a fdsn station service v. 2.0, thus under the URL fdsnws/station/2/ .
I suggest that the FDSN as holder of the standard quickly declares this new version and avoids all ambiguity before heterogeneous treatment of the issue starts to creep into the worldwide population of fdsn station services.
Kind regards,
Philipp
********************
Dear Jean-Marie,
With https://www.xml.com/pub/a/2003/12/03/versioning.html (1), you actually point to an excellent read.
However before discussing its implication, I have to clarify on the term *compatibility*, as the document (1) uses it in a different way.
(1) describes a SOAP context, where a customer sends an XML instruction to a service, which in return does something. Here, a backwards compatible service means that the new service version (implemented for digesting the new version of the instruction xml) is still capable to interpret the old one. Technically it means that the old xml fulfills also the new schema. Usability-wise (and I think this is the key point) it means that the new service version does not break the user expectation of a customer who is not aware of the service version change.
(1) fdsnws (https://www.fdsn.org/webservices/FDSN-WS-Specification-Commonalities-1.2.pdf) describes a REST context, where a customer sends a GET request and, in return, receives a stationXML. A backwards compatible service, from a customer's perspective, means that a change of the service fulfills both requirements a) the service can understand an unchanged get request (a get request of a customer which is not aware of the new minor version, or implementation of the service), and b) the service returns a result a client unaware of the new service minor version or implementation "can understand". "can understand", in the context of a standard defined in xsd, means: it fulfills (validates against) the unchanged (old) schema.
Only if a) and b) are met, a customer can go on using the updated service without being aware of the update. (thus without the need from the service provider's side to announce the change, and give the customer time for adaptation.
Now I admit that this definition of compatibility relies more on common sense than on a literature source. However it seems in line with the intention of the authors of the FDSNws standard: they requested that a not compatible = major service upgrade should be made transparent to the customer by a change of the URL (more precisely, the major version number in the URL). A changed URL is a very powerful way to tell the user of the old URL that things are not as they were.
Using this definition, an fdsn/station service returning StationXML 1.1 documents must have a different major version than its precedecessor returning StationXML 1.0, *if* a StationXML 1.1 *document* does *not* validate against the StationXML 1.0 *schema* definition (the one used by the update-unaware client to validate the response). Not vice versa.
Now, stationXML 1.0 and 1.1 have the same namespace, which makes the same schemata at least applicable, and the changing version number does not per se break validation. So the question why a stationXML 1.1 document is not, at the same time, valid stationXML 1.0 is a justified one.
It is not. XML works not in a way that you can, by default, just add anything anywhere, and if it is unknown to the schema, it is ignored. But you are required to define where "something unknown" can be added, and you can restrict what "unknown" means. This has been done in stationXML 1.0 - look in the schema for <xs:any> for any element, and xs:anyattribute for any attribute. However, you will see that all these extension points have the namespace="##other" attribute, thus these random add-ons to stationXML documents can be *anything but* elemnts defined in the stationXML scheme (why? To avoid ambiguity**; see below). As the new elements added in stationXML 1.1 are (obviously) part of stationXML, they do not qualify as "any unknown add-on" as allowed according to the 1.0 scheme.
So all this is no argument against stationXML 1.1, or fdsn station web services. It is just an argument to offer stationXML 1.1 only on an fdsnws/station/2/query? service url, in order to make sure the customer is aware of the fact that he gets something he/she has to treat technically differently from what he gets from an fdsnws/station/1/query? service URL.
I posted this issue on the FDSN WG 3 mailing group first as they are holders of both the stationXML and the fdsnws specifications, and should provide a fdsnws/station/2/ specification document with clarification on the topic. Although the practical implications are probably largest in Europe (aka EIDA), where scientists have to deal with many different service endpoints, potentially updated at a different pace...
Best regards,
Philipp
***********
** on ambiguity: imagine you have an staxml:station element type with a sequence of staxml:latitude (minOccurs: 1, maxOccurs: 1), a staxml:longitude (minOccurs:1, maxOccurs: 1), and xs:any (minOccurs:0, maxOccurs: indefinite) child elements. Now, if you allow the xs:any to be *really* anything (including elements from the staxml scheme), a user could do this:
<station code="BLA">
<latitude>47.0</latitude>
<longitude> 7.0</longitude>
<latitude>48.0</latitude>
</station>
, which is, at the same time valid and not valid, and it is definitely not what you would like to have passed to a staxml-aware software code...
-----Original Message-----
From: eida_maint<at>gfz-potsdam.de [eida_maint<at>gfz-potsdam.de]
Sent: Donnerstag, 5. Dezember 2019 11:52
To: eida_maint<at>gfz-potsdam.de
Subject: Re: fdsnsws-station and stationXML1.1
Dear Philipp,
Thanks for those clarifications.
I'm also working on stationXML outside RESIF, I let them answer from their perspective.
And since I will have, among other things, polynomial responses, I must use 1.1 version.
* Anyway, you might have understood I'm far from an XML expert.
I previously understood that any tag addition in an XML file (I'm not speaking specifically about stationXML here) was not a problem as clients can be made to only read a subset of them and ignore any unknown tag.
Which was perfectly fine for me.
And from your answer, I'm puzzled because I understand that any tag addition will break the backward compatibility.
This would mean we would never be able to do any revision of an XML standard without breaking the compatibility.
* Could it be because you are trying to validate a 1.1 document with regards to a 1.0 standard ?
This obviously will break because, as it is written in the stationXML header, it's a 1.1 version, not a 1.0 version.
* Quickly browsing through the web, the XML standard has an "Ignore Rule" that says precisely that a client "Must ignore" an unknown tag.
https://www.xml.com/pub/a/2003/12/03/versioning.html
https://www.goland.org/xmlschemabackwardscompat/
Is that wrong ?
Regards.
Jean-Marie.
Le 05/12/2019 à 09:35, Kästli Philipp kaestli<at>sed.ethz.ch a écrit :
Dear Jean-Marie,
--
that stationXML 1.1 breaks compatibility with stationXML 1.1 you can
verify easily by actually validating an 1.1 document against the
stationXML 1.0 schem:
https://www.fdsn.org/xml/station/fdsn-station-1.0.xsd
nonix:~/eida/stationxml$ curl
'https://www.fdsn.org/xml/station/fdsn-station-1.0.xsd' >
fdsn-station-1.0.xsd nonix:~/eida/stationxml$ curl
'https://ws.resif.fr/fdsnws/station/1/query?network=G&station=ECH&leve
l=station&format=xml'>
resif.xml
xmllint --schema fdsn-station-1.0.xsd resif.xml <?xml version="1.0"
encoding="UTF-8"?> <FDSNStationXML
xmlns="http://www.fdsn.org/xml/station/1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:resif="http://www.fdsn.org/xml/station/1/resif"
schemaVersion="1.1"
xsi:schemaLocation="http://www.fdsn.org/xml/station/1
http://www.fdsn.org/xml/station/fdsn-station-1.1.xsd">
[...]
</FDSNStationXML>
resif.xml:7: element Identifier: Schemas validity error : Element
'{http://www.fdsn.org/xml/station/1}Identifier': This element is not
expected. Expected is one of (
{http://www.fdsn.org/xml/station/1}Comment,
##other{http://www.fdsn.org/xml/station/1}*,
{http://www.fdsn.org/xml/station/1}TotalNumberStations,
{http://www.fdsn.org/xml/station/1}SelectedNumberStations,
{http://www.fdsn.org/xml/station/1}Station ).
resif.xml fails to validate
nonix:~/eida/stationxml$
XML as a format family is generally chosen because it is defined which
tags must, must can be present (and which tag contents are valid), and
bhus, because an xml document can easily be verified for validity
against the respective schema.In an xml-based environment, the
possibility to validate is a must.
Even if you try to cheat (e.g. by declaring the 1.1 document as 1.0
and/or skipping validation), any client software generated against the
stationXML 1.0 scheme will fail to handle it, e.g. as the XML document
makes use of the stationXML 1.1 option that any BaseNode type (station
and channel) can have a persistent identifier (which was given in
stationXML 1.0), or if it lacks a creationDate (which is allowed to
lack in 1.1, but forbidden to lack in 1.0), if it has an operator at
the network level (allowed in 1.1, not allowed in 1.0), if it has a
subject or lacks an id in a comment (both not allowed in 1.0), or similar cases.
For the full changes list, see
https://www.fdsn.org/xml/station/fdsn-station-changes-1.0-to-1.1.txt
Kind regards, Philipp
----------------------------------------------------------------------
--
*From:* eida_maint<at>gfz-potsdam.de <eida_maint<at>gfz-potsdam.de> on
behalf of Jean-Marie SAUREL saurel<at>ipgp.fr <eida_maint<at>gfz-potsdam.de>
*Sent:* 05 December 2019 09:47
*To:* eida_maint<at>gfz-potsdam.de
*Subject:* Re: fdsnsws-station and stationXML1.1 Good morning
everyone,
I also don't clearly understand why the stationXML 1.1 breaks
compatibility with stationXML 1.0.
Do you have any more details on that ?
The only item that stationXML 1.1 removed is the StorageFormat tag,
which, as it was pointed in FDSN WG3, is very unlikely to have any use.
Furthermore, this tag is not a mandatory tag according to stationXML
1.0 xsd (minoccurs=0).
The other main modification is the correction of an error in
stationXML
1.0 polynomial response representation.
The 1.0 version request a StageGain to be present event in a
polynomial response, which simply makes no sense and is against SEED manual 2.4.
The only way I imagine a 1.1 stationXML could break a client coded for
1.0 would be because of this missing StageGain in polynomial response.
Otherwise, a correctly coded stationXML 1.0 should not break if there
are optional or non-mandatory missing tags.
Regards.
Jean-Marie.
Le 05/12/2019 à 08:30, Jonathan Schaeffer
jonathan.schaeffer<at>univ-grenoble-alpes.fr a écrit :
Hi John and all,
--
I'd like to emphasize that RESIF does not break any specification of
fdsnws:station1.0 (as you said).
If we (ETC) are to identify the best options, we need to understand
why
stationXML1.1 "breaks any clients".
We did not any complaint from users using Obspy as far as I know.
The suggestion (having fdsnws:station2.0 and fdsnws:station1.2
specifications and implementations) for this minor change will be a
real pain for everyone, and in particular for the end users. I see a
lot of usage drawbacks and won't elaborate here.
If this issue is an emergency, I suggest we plan a visio-conference
of ETC on this point. If it can wait untill the next ETC meeting,
let's put it on the agenda.
Cheers,
Jonathan
On 04/12/2019 22:07, Clinton John jclinton<at>sed.ethz.ch wrote:
Dear EMB and ETC,
You may recall a new version of stationXML was released by FDSN in
May last year, a minor version increase from 1.0 to 1.1.
A few days ago, most of you will have noticed an email from Philipp
and Daniel to the FDSN WG3. They spotted that the RESIF node updated
their fdsnws-station to provide XML in stationXMl1.1 format.
Currently, we do not find any other fdsnws-event service that
returns stationXML1.1, even outside of EIDA. Inside EIDA, this
inconsistency, and lack of control on the format received from
fdsnws-station, breaks any clients, including the federator, and
will cause downstream issues for other users. For example, now a
simple request for metadata from AlpArray Z3 network will provide information in both formats.
Unfortunately, there was no migration plan for the corresponding
fdsnws-station service - the specification is loose and 'allows'
distribution of different minor versions. As I understand, the
situation with different EIDA nodes providing different versions of
stationXML from the same service is not acceptable in the long term.
A proposed plan that 1) ensures immediate re-compliance across EIDA
and
2) provides a definite path to all migrate to the latest stationXML
version could be:
- RESIF reverts back to releasing stationXML1.0 from
fdsnws-station1.1 as soon as possible.
*/(this unblocks the Federator developments)/*
- EIDA requests to FDSN WG3 that
a minor version fdsnws-station1.2 is released, clearly specifies it
returns stationXML1.0 format a major version fdsnws:station2.0, is
released, clearly specifies it returns stationXML1.1 format
- An engineering phase is required to ensure all nodes can provide
both
fdsnws:station1.2 and 2.0. For nodes that provide fdsnws via SC3,
this requires updating the software, most reasonably by gempa,
followed by installation of the new SC3 version at all nodes.
- all EIDA nodes would then support both
fdsnws:station1.2 to deliver stationXML1.0
fdsnws:station2.0 to deliver stationXML1.1 */(then the Federator can
also support stationXML1.0 or 1.1, depending on configuration of the
routing service)/*
I have discussed the issue with Helle today. She notes RESIF is not
currently in a position to provide feedback or take actions for a
couple of weeks and may not actually be able to revert to stationXML1.0.
Further, she notes this plan is relatively complex and may take
time - which is true. On the other hand, this is an opportunity for
EIDA to insist the fdsnws definitions become tighter and more
precise, and to point out future format changes need to be carefully
coupled with sufficient and appropriate changes to exchange formats.
In the meantime, I would really appreciate opinions from the EIDA nodes.
Would others agree to the proposed plan, or suggest another plan or
even an entirely different strategy? Ideally, we would be able to
quickly come to a consensus, approach WG3 and speak on behalf of
EIDA on this issue If needed, we at ETH are happy to organise a
conference call with interested parties on this issue.
Regards,
John
*From:* Kästli Philipp
*Sent:* Donnerstag, 28. November 2019 16:01
*To:* FDSN Working Group III <fdsn-wg3-products<at>lists.fdsn.org
<fdsn-wg3-products<at>lists.fdsn.org>>
*Subject:* stationXML version 1.1 and fdsnws/station service
version 1.x
Dear colleagues of FDSN WG3,
Our last fdsnws/station specification is 1.2 and dates
(content-wise) of April 2, 2019 (an update in June refers to the
formatting of the standards document only).
According to this document, the fdsnws/station response format is
"StationXML" (or text), without indication of the version, and
thus, implicitly, referring to StationXML 1.0
On May 3, 2019, FDSNwg 3 published a new version 1.1 of StationXML.
John and Chad describe the the compatibility situation as follows:
"Additionally, a number of backwards-compatible changes suggested
at previous meetings and on the project site were incorporated.
Conforming to the versioning rules, all 1.0 version documents are
also compliant with schema version 1.1, with one exception: the
<StorageFormat> element was removed from the schema, but is not
widely used and not expected to cause incompatibility."
We would summarize this as follows:
- stationXML 1.0 remains forward-compatible with 1.0, thus
an "old" xml document is still readable for software implemented
for the
1.1 format (with the exception mentioned)
- stationXML 1-1 is *not* backward-compatible with 1.1,
thus a new document violates the old specification and is not
readable for software implemented against the 1.0 specification.
This has implications on the fdsnws/station/1/ web service. The
standards document v. 1.2 says the following on service versioning:
"Web service implementations are versioned according the following
three-digit (x.y.z) pattern:
SpecMajor . SpecMinor . Implementation Where the fields have the
following meaning:
SpecMajor : The major specification version*, all implementations
sharing this SpecMajor value will be backwards compatible with all
prior releases.* Values start at 1. " And "The following base URI
pattern is to be used at each data center implementing FDSN
services: <site>/fdsnws/<service>/*<majorversion>*/"
Thus, an FDSNWS user can expect that if she/he has implemented an
fdsnws-station client according to the standard (thus, capable of
digesting the stationXML of the time, e.g. 1.0), this client will
never break, as long as she/he addresses the service under the same
url ./fdsnws/station/*1*/. .
An updated service with a non-backward-compatible response should
have a different url ./fdsnws/station/2/. So, at the time being, it
violates the fdsnws/station v. 1.x standard, and the justified
expectation of some users, if a service fdsnws/station/1/ 1.x
service returns the new stationXML 1.1 format.
(at the same time, an fdsnws/station service provider of 2020 may,
when reading the fdsnws/station/1/ specification, may not pay
attention that it pre-dates the stationXML 1.1 version, and
consider returning that very format)
Proposed way out:
- have a new fdsnws/station/1/ specification version
1.2(.1)
with the only change that it explicitly says that it is referring
to stationXML 1.0 as its response format
- have a fdsnws/station/2/ specification 2.0 saying that
the response format is 1.1 or any *backward* compatible upper
version
Kind regards,
Daniel Armbruster
Philipp Kästli
Swiss Seismological Service @ ETHZ
--------------------------------------
Institut de Physique du Globe de Paris Observatoires Volcanologiques
et Sismologiques
+33 1 83 95 74 37
+33 6 20 35 28 04 (portable)
1 rue Jussieu
75238 Paris cédex 5
#############################################################
This message is sent to you because you are subscribed to
the mailing list <eida_maint<at>gfz-potsdam.de>.
To unsubscribe, E-mail to: <eida_maint-off<at>gfz-potsdam.de> To switch
to the DIGEST mode, E-mail to <eida_maint-digest<at>gfz-potsdam.de>
To switch to the INDEX mode, E-mail to
<eida_maint-index<at>gfz-potsdam.de> Send administrative queries to
<eida_maint-request<at>gfz-potsdam.de>
--------------------------------------
Institut de Physique du Globe de Paris
Observatoires Volcanologiques et Sismologiques
+33 1 83 95 74 37
+33 6 20 35 28 04 (portable)
1 rue Jussieu
75238 Paris cédex 5
#############################################################
This message is sent to you because you are subscribed to
the mailing list <eida_maint<at>gfz-potsdam.de>.
To unsubscribe, E-mail to: <eida_maint-off<at>gfz-potsdam.de> To switch to the DIGEST mode, E-mail to <eida_maint-digest<at>gfz-potsdam.de>
To switch to the INDEX mode, E-mail to <eida_maint-index<at>gfz-potsdam.de> Send administrative queries to <eida_maint-request<at>gfz-potsdam.de>
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
Hi
Just want to point out that the stationxml contains the version of the schema it corresponds to, in the schemaVersion attribute and in the location of the schema for validation in the schemaLocation attribute, so a well-coded client can tell a 1.0 from a 1.1 xml file and act appropriately.
Are you aware of a specific client and service/url combination that breaks because of this? My memory is that the reason this exception to the versioning rule was allowed was that there were no known instances where the items changed were used, and so it was felt safe to allow it.
Before creating a new version of the web services, it is probably worth investigating whether or not the concern is an actual problem occurring in the real world or not. In general I am in favor of following semver, but the large disruption of a fdsn station url change has to be weighed against the benefits.
Philip
-
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully parse the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
Dear WG 3 members,
To move this issue forward I ask that members share their opinions and suggested solutions by the end of February. This will allow us to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that contain elements/attributes not defined in the 1.0 schema cannot be validated using the 1.0 schema as they (purposely) use the same namespace and 1.0 does not allow additions in the same namespace. A conflict may occur, for example, when a client using an fdsnws-station 1.x service receives StationXML 1.1 that contains an <Identifier> element (new to 1.1) and tries to validate the response against the StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients expecting version 1.0 (ignoring the new elements/attributes), which works in practice. Validation of a 1.1 document using the 1.0 schema is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station service that would have a new endpoint. This would remove the ambiguity of which version of StationXML is returned (by default or via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML 1.x, and that any clients that wish to validate the responses must use the matching, or latest 1.x, schema specification. Note: the schema specification version is declared in the document itself, often with a link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be released with modifications that allow the documents to be validated with the 1.0 schema; then deprecate schema version 1.1 (not yet widely adopted) and move quickly to 1.2. I do not know of a technical solution for this, but it may be possible.
If you know of other options please share them to be considered. If we determine that the issue is best solved in a StationXML schema change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and fdsnws-station specifications are likely. I recommend that we determine a clear path that allows for these releases with minimal disruption and invite suggestions for how to accomplish this.
regards,
Chad
On Dec 6, 2019, at 12:32 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch> wrote:
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully parse !
the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
Hi
I am not sure how I feel about this issue forcing a version 2 of the
web services spec, but if there is a new major version, ie
/fdsnws/station/2, perhaps adding a "schema" endpoint would prevent
this issue in the future. The idea being that a version >=2 client is
expected to validate the stationxml returned from
/fdsnws/station/2/query against the schema returned from
/fdsnws/station/2/schema at the same datacenter. The client should not
validate against a predefined schema. This provides an easy way to get
the correct schema, even though in practice it is also available at
the URL in the stationxml document.
Allowing an optional "format=" query parameter on this service would
also allow schema (if they exist) for other formats, like geojson or
kml that are returned by some data centers.
At this point we can then define non-breaking change, which allows a
minor version bump, as "designed to be readable and usable by clients
expecting version x.0 so long as they ignore the new
elements/attributes" but specifically does not expect validation
against the older x.0 schema. Validation should only occur against the
schema that corresponds to the version, from either the schema
endpoint, or the corresponding schema url in the xml document (both
assumed to be equivalent).
Without something like this, I suspect that no change can ever be
considered applicable for a minor version, which implies only major
versions and hence there is never any backward compatibility.
Philipp's federator example, would then validate the stationxml from
the various datacenters using the schema that they expect to be used.
The merging would of course require intelligence to know how to
combine 1.0 and 1.1 stationxml documents, but that likely is needed
regardless. I would guess that the federator should use the highest
version known and be able to map earlier versions.
Philip
On Mon, Feb 3, 2020 at 6:58 PM Chad Trabant <chad<at>iris.washington.edu> wrote:
Dear WG 3 members,
To move this issue forward I ask that members share their opinions and suggested solutions by the end of February. This will allow us to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that contain elements/attributes not defined in the 1.0 schema cannot be validated using the 1.0 schema as they (purposely) use the same namespace and 1.0 does not allow additions in the same namespace. A conflict may occur, for example, when a client using an fdsnws-station 1.x service receives StationXML 1.1 that contains an <Identifier> element (new to 1.1) and tries to validate the response against the StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients expecting version 1.0 (ignoring the new elements/attributes), which works in practice. Validation of a 1.1 document using the 1.0 schema is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station service that would have a new endpoint. This would remove the ambiguity of which version of StationXML is returned (by default or via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML 1.x, and that any clients that wish to validate the responses must use the matching, or latest 1.x, schema specification. Note: the schema specification version is declared in the document itself, often with a link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be released with modifications that allow the documents to be validated with the 1.0 schema; then deprecate schema version 1.1 (not yet widely adopted) and move quickly to 1.2. I do not know of a technical solution for this, but it may be possible.
If you know of other options please share them to be considered. If we determine that the issue is best solved in a StationXML schema change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and fdsnws-station specifications are likely. I recommend that we determine a clear path that allows for these releases with minimal disruption and invite suggestions for how to accomplish this.
regards,
Chad
On Dec 6, 2019, at 12:32 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch> wrote:
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully parse !
the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
If schema validators were able to follow XML Schema 1.1 conventions, there's the notion of a Schema Versioning Workspace that allows for selective version ranges on individual elements. This practice should probably only be followed for minor version updates, as you can continue to use the same StationXML namespace (/1) with progressive updates. A major version is not meant to be backward or forward compatible, so the StationXML namespace would change major versions...and I would think the service endpoint as well.
Here's the W3C doc on the vc namespace:
https://www.w3.org/TR/xmlschema11-1/#vc-namespace
If this were something the working group wanted to pursue, the vc namespace could be introduced to a 1.2 version of the doc, with the focus of 'forward-compatibility' for older clients to be compatible on 1.0, 1.1, and 1.2 docs.
-Rob
On Feb 4, 2020, at 8:48 AM, Philip Crotwell <crotwell<at>seis.sc.edu> wrote:
Hi
I am not sure how I feel about this issue forcing a version 2 of the
web services spec, but if there is a new major version, ie
/fdsnws/station/2, perhaps adding a "schema" endpoint would prevent
this issue in the future. The idea being that a version >=2 client is
expected to validate the stationxml returned from
/fdsnws/station/2/query against the schema returned from
/fdsnws/station/2/schema at the same datacenter. The client should not
validate against a predefined schema. This provides an easy way to get
the correct schema, even though in practice it is also available at
the URL in the stationxml document.
Allowing an optional "format=" query parameter on this service would
also allow schema (if they exist) for other formats, like geojson or
kml that are returned by some data centers.
At this point we can then define non-breaking change, which allows a
minor version bump, as "designed to be readable and usable by clients
expecting version x.0 so long as they ignore the new
elements/attributes" but specifically does not expect validation
against the older x.0 schema. Validation should only occur against the
schema that corresponds to the version, from either the schema
endpoint, or the corresponding schema url in the xml document (both
assumed to be equivalent).
Without something like this, I suspect that no change can ever be
considered applicable for a minor version, which implies only major
versions and hence there is never any backward compatibility.
Philipp's federator example, would then validate the stationxml from
the various datacenters using the schema that they expect to be used.
The merging would of course require intelligence to know how to
combine 1.0 and 1.1 stationxml documents, but that likely is needed
regardless. I would guess that the federator should use the highest
version known and be able to map earlier versions.
Philip
On Mon, Feb 3, 2020 at 6:58 PM Chad Trabant <chad<at>iris.washington.edu> wrote:
Dear WG 3 members,
e !
To move this issue forward I ask that members share their opinions and suggested solutions by the end of February. This will allow us to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that contain elements/attributes not defined in the 1.0 schema cannot be validated using the 1.0 schema as they (purposely) use the same namespace and 1.0 does not allow additions in the same namespace. A conflict may occur, for example, when a client using an fdsnws-station 1.x service receives StationXML 1.1 that contains an <Identifier> element (new to 1.1) and tries to validate the response against the StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients expecting version 1.0 (ignoring the new elements/attributes), which works in practice. Validation of a 1.1 document using the 1.0 schema is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station service that would have a new endpoint. This would remove the ambiguity of which version of StationXML is returned (by default or via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML 1.x, and that any clients that wish to validate the responses must use the matching, or latest 1.x, schema specification. Note: the schema specification version is declared in the document itself, often with a link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be released with modifications that allow the documents to be validated with the 1.0 schema; then deprecate schema version 1.1 (not yet widely adopted) and move quickly to 1.2. I do not know of a technical solution for this, but it may be possible.
If you know of other options please share them to be considered. If we determine that the issue is best solved in a StationXML schema change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and fdsnws-station specifications are likely. I recommend that we determine a clear path that allows for these releases with minimal disruption and invite suggestions for how to accomplish this.
regards,
Chad
On Dec 6, 2019, at 12:32 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch> wrote:
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully pars!
the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
----------------------
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
-
The Station XML 1.0 schema appears to allow elements and attributes from any namespace.
One option is to use the existing namespace where elements are unchanged, and use a different namespace for Station XML 1.x elements. This maintains backward compatibility for existing 1.0 parsers, while allowing 1.x aware parsers to read the additional information.
Thanks,
Jeremy Fee
Computer Scientist
USGS Geologic Hazards Science Center, Golden, CO
________________________________
From: fdsn-wg3-products-bounce<at>fdsn.org <fdsn-wg3-products-bounce<at>fdsn.org> on behalf of Chad Trabant <chad<at>iris.washington.edu>
Sent: Monday, February 3, 2020 16:57
To: FDSN Working Group III <fdsn-wg3-products<at>lists.fdsn.org>
Subject: [EXTERNAL] Re: [fdsn-wg3-products] fdsnsws-station and stationXML1.1
Dear WG 3 members,
To move this issue forward I ask that members share their opinions and suggested solutions by the end of February. This will allow us to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that contain elements/attributes not defined in the 1.0 schema cannot be validated using the 1.0 schema as they (purposely) use the same namespace and 1.0 does not allow additions in the same namespace. A conflict may occur, for example, when a client using an fdsnws-station 1.x service receives StationXML 1.1 that contains an <Identifier> element (new to 1.1) and tries to validate the response against the StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients expecting version 1.0 (ignoring the new elements/attributes), which works in practice. Validation of a 1.1 document using the 1.0 schema is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station service that would have a new endpoint. This would remove the ambiguity of which version of StationXML is returned (by default or via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML 1.x, and that any clients that wish to validate the responses must use the matching, or latest 1.x, schema specification. Note: the schema specification version is declared in the document itself, often with a link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be released with modifications that allow the documents to be validated with the 1.0 schema; then deprecate schema version 1.1 (not yet widely adopted) and move quickly to 1.2. I do not know of a technical solution for this, but it may be possible.
If you know of other options please share them to be considered. If we determine that the issue is best solved in a StationXML schema change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and fdsnws-station specifications are likely. I recommend that we determine a clear path that allows for these releases with minimal disruption and invite suggestions for how to accomplish this.
regards,
Chad
On Dec 6, 2019, at 12:32 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch<kaestli<at>sed.ethz.ch>> wrote:
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully parse !
the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org<fdsn-wg3-products-unsubscribe<at>lists.fdsn.org>
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org<fdsn-wg3-products-unsubscribe<at>lists.fdsn.org>
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
The Station XML 1.0 schema appears to allow elements and attributes from any namespace.
Yes, any additional element or attribute from any namespace except its own.
One option is to use the existing namespace where elements are unchanged, and use a different namespace for Station XML 1.x elements.
So, technically, all add-ons of current stationXML 1.1 compared to 1.0 could be implemented with such an extension to stationxml 1(.0) in a separate namespace. However, as far as I know, you cannot introduce restrictions to elements in the stationXML namespace from a different namespace.
This maintains backward compatibility for existing 1.0 parsers, while allowing 1.x aware parsers to read the additional information.
Thus, the following stationXML 1.1 features / restrictions, could probably not be enforced with an extension in a different namespace:
* disallow "unit" attribute to <Numerator> and <Denominator>
* Allow <CreationDate> to be optional (change from required)
* Use xs:double for <ApproximationLowerBound>, <ApproximationUpperBound> and <MaximumError>
* Include data availability elements described in the fdsn-station+availability-1.0.xsd extension schema as optional elements of the main schema (they would stay in a different namespace)
* Remove <StorageFormat> from <Channel> (i.e. somebody can decide not to add the storage format, but an xml containing it remains valid)
* Limit each <Operator> to a single <Agency>
* Allow more than a single <Equipment> occurrence in <Channel>, same as in <Station>
* Do not require and disallow <StageGain> and <Decimation> for <Polynomial> response stages
So in my understanding, and referring to Chad's proposal #3, such an stationXML 1.2 with 1.1 features, but 1.0 compatibility is only partly possible.
Cheers, Philipp
________________________________
From: fdsn-wg3-products-bounce<at>fdsn.org<fdsn-wg3-products-bounce<at>fdsn.org> <fdsn-wg3-products-bounce<at>fdsn.org<fdsn-wg3-products-bounce<at>fdsn.org>> on behalf of Chad Trabant <chad<at>iris.washington.edu<chad<at>iris.washington.edu>>
Sent: Monday, February 3, 2020 16:57
To: FDSN Working Group III <fdsn-wg3-products<at>lists.fdsn.org<fdsn-wg3-products<at>lists.fdsn.org>>
Subject: [EXTERNAL] Re: [fdsn-wg3-products] fdsnsws-station and stationXML1.1
Dear WG 3 members,
To move this issue forward I ask that members share their opinions and suggested solutions by the end of February. This will allow us to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that contain elements/attributes not defined in the 1.0 schema cannot be validated using the 1.0 schema as they (purposely) use the same namespace and 1.0 does not allow additions in the same namespace. A conflict may occur, for example, when a client using an fdsnws-station 1.x service receives StationXML 1.1 that contains an <Identifier> element (new to 1.1) and tries to validate the response against the StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients expecting version 1.0 (ignoring the new elements/attributes), which works in practice. Validation of a 1.1 document using the 1.0 schema is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station service that would have a new endpoint. This would remove the ambiguity of which version of StationXML is returned (by default or via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML 1.x, and that any clients that wish to validate the responses must use the matching, or latest 1.x, schema specification. Note: the schema specification version is declared in the document itself, often with a link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be released with modifications that allow the documents to be validated with the 1.0 schema; then deprecate schema version 1.1 (not yet widely adopted) and move quickly to 1.2. I do not know of a technical solution for this, but it may be possible.
If you know of other options please share them to be considered. If we determine that the issue is best solved in a StationXML schema change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and fdsnws-station specifications are likely. I recommend that we determine a clear path that allows for these releases with minimal disruption and invite suggestions for how to accomplish this.
regards,
Chad
On Dec 6, 2019, at 12:32 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch<kaestli<at>sed.ethz.ch>> wrote:
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully parse !
the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org<fdsn-wg3-products-unsubscribe<at>lists.fdsn.org>
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org<fdsn-wg3-products-unsubscribe<at>lists.fdsn.org>
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
-
On 04/02/2020 00:57, Chad Trabant wrote:
Dear WG 3 members,
Hi everyone,
To move this issue forward I ask that *members share their opinions*
*and suggested solutions* *by the end of February*. This will allow us
to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that
contain elements/attributes not defined in the 1.0 schema cannot be
validated using the 1.0 schema as they (purposely) use the same
namespace and 1.0 does not allow additions in the same namespace. A
conflict may occur, for example, when a client using an fdsnws-station
1.x service receives StationXML 1.1 that contains an <Identifier>
element (new to 1.1) and tries to validate the response against the
StationXML 1.0 schema.
sorry if my suggestion is actually stupid, but I'm not aware of all the
possible use-cases. So please don't hurt mie, I still wanted to suggest
this :
And what about taking another approach : let the datacenters be
responsible for the syntaxic validity of the metadata delivered instead
of the clients.
We (datacenters) only need to announce in the metadata wich version it
is (stationXML1.x), and the clients work with that. They don't need to
validate, they just have to handle the differences between the minor
versions *if they need to*.
Thus, no need for a new major version of stationXML service.
Does this solve anything ?
regards,
--
Jonathan Schaeffer
Observatoire des Sciences de l'Univers de Grenoble
Responsable Infrastructure RESIF-DC
✆ +33 4 76 63 51 37 ou +33 4 76 63 52 48
🏢 Isterre, bureau 035
1681 rue de la Piscine
38400 Grenoble Cedex
Attachments
-
cc: EIDA Management Board
Dear Chad, dear all,
Taking into account potential costs and delays, the 11 EIDA nodes
(ODC/KNMI, GFZ, RESIF, INGV, ETHZ, BGR, NIEP, KOERI, NOA, UIB, LMU) do
not support the immediate creation of new versions for either stationXML
or fdsnws-station. Our standpoint is motivated by concerns that any
additional format/specification changes will lead to undue delays that
will compound current issues within EIDA where we have different
stationXML output across EIDA nodes.
Our plan is therefore to migrate all EIDA nodes to deliver stationXML1.1
as soon as possible.
Note that although all EIDA nodes intend to migrate fast to an internal
solution that has all nodes supporting a single version of stationXML,
the issues concerning ambiguous fdsnws specifications as well as other
fdsnws-station implementations providing different XML versions will
persist. In the long run, this should be addressed within future format
/ specification changes.
Please notify us if the FDSN needs individual statements from each node
Best regards
Helle Pedersen
Chair of the EIDA Management Board.
Le 04/02/2020 à 00:57, Chad Trabant a écrit :
Dear WG 3 members,
--
To move this issue forward I ask that *members share their opinions*
*and suggested solutions* *by the end of February*. This will allow us
to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that
contain elements/attributes not defined in the 1.0 schema cannot be
validated using the 1.0 schema as they (purposely) use the same
namespace and 1.0 does not allow additions in the same namespace. A
conflict may occur, for example, when a client using an fdsnws-station
1.x service receives StationXML 1.1 that contains an <Identifier>
element (new to 1.1) and tries to validate the response against the
StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients
expecting version 1.0 (ignoring the new elements/attributes), which
works in practice. Validation of a 1.1 document using the 1.0 schema
is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station
service that would have a new endpoint. This would remove the
ambiguity of which version of StationXML is returned (by default or
via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML
1.x, and that any clients that wish to validate the responses must use
the matching, or latest 1.x, schema specification. Note: the schema
specification version is declared in the document itself, often with a
link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be
released with modifications that allow the documents to be validated
with the 1.0 schema; then deprecate schema version 1.1 (not yet widely
adopted) and move quickly to 1.2. I do not know of a technical
solution for this, but it may be possible.
If you know of other options please share them to be considered. If
we determine that the issue is best solved in a StationXML schema
change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and
fdsnws-station specifications are likely. I recommend that we
determine a clear path that allows for these releases with minimal
disruption and invite suggestions for how to accomplish this.
regards,
Chad
Helle Pedersen
ISTerre
Université Grenoble Alpes
CS 40700
38058 GRENOBLE Cedex 9
FRANCE
Tel +33 (0)4 76 63 52 59
Fax +33 (0)4 76 63 52 01
-
Dear WG III members,
Thank you to those that contributed your thoughts on this issue. Based on responses, consensus among members is not to release a new specification for the fdsnws-station service and, instead to encourage the service providers to migrate their implementations to offer StationXML 1.1 as quickly as possible.
We also agree with the sentiment expressed by multiple members that future versions of the web service specifications and the StationXML format should be designed to avoid the ambiguity identified in this case moving forward.
regards,
Chad Trabant (WG III Chair)
Mark Chadwick (WG III Vice Chair)
On Feb 3, 2020, at 4:00 PM, Chad Trabant <chad<at>iris.washington.edu> wrote:
Dear WG 3 members,
To move this issue forward I ask that members share their opinions and suggested solutions by the end of February. This will allow us to determine if quick action by the WG is needed.
To summarize the issue as understand it: StationXML 1.1 documents that contain elements/attributes not defined in the 1.0 schema cannot be validated using the 1.0 schema as they (purposely) use the same namespace and 1.0 does not allow additions in the same namespace. A conflict may occur, for example, when a client using an fdsnws-station 1.x service receives StationXML 1.1 that contains an <Identifier> element (new to 1.1) and tries to validate the response against the StationXML 1.0 schema.
Note: StationXML 1.1 was designed to be readable and usable by clients expecting version 1.0 (ignoring the new elements/attributes), which works in practice. Validation of a 1.1 document using the 1.0 schema is the case that does not work.
I see a couple of options to consider:
1) Define a new, major version 2, specification of the fdsnws-station service that would have a new endpoint. This would remove the ambiguity of which version of StationXML is returned (by default or via the format="xml") selection.
2) The FDSN could declare that fdsnws-station 1.x returns StationXML 1.x, and that any clients that wish to validate the responses must use the matching, or latest 1.x, schema specification. Note: the schema specification version is declared in the document itself, often with a link to the appropriate schema.
3) The next minor version of the StationXML schema (1.2) could be released with modifications that allow the documents to be validated with the 1.0 schema; then deprecate schema version 1.1 (not yet widely adopted) and move quickly to 1.2. I do not know of a technical solution for this, but it may be possible.
If you know of other options please share them to be considered. If we determine that the issue is best solved in a StationXML schema change, it will be referred to WG 2.
We should be mindful that future, minor releases of the StationXML and fdsnws-station specifications are likely. I recommend that we determine a clear path that allows for these releases with minimal disruption and invite suggestions for how to accomplish this.
regards,
Chad
On Dec 6, 2019, at 12:32 AM, Philipp Kästli @fdsn.org <kaestli<at>sed.ethz.ch <kaestli<at>sed.ethz.ch>> wrote:
Dear Philip
Are you aware of a specific client and service/url combination that breaks because of this? My memory is
I guess any decent client of a service providing a document in a format defined by an xsd would validate the document against that xsd in order to make sure that the subsequent processing software can cope with its content - it is probably the strongest reason why to use xml formats at all.
that the reason this exception to the versioning rule was allowed was that there were no known instances > where the items changed were used, and so it was felt safe to allow it.
To point to a specific one: In Europe, we have many "small" data centers providing data of one or a few of our small countries (often entire networks, but some networks are also distributed over multiple datacenters). In order to make the life of the scientists easier, EIDA develops a "federator", i.e. a software which offers an fdsnws at its frontend from where customers can request (meta)data e.g. of areas covering many countries. The federator transparently & "intelligently" redistributes this request to all national or regional data centers, collects and merges the responses, and forwards the joint result to the customer. In a service landscape where the national services can return on the same request /fdsnws/station/1/query?...&format=xml technically (on the level of validation) different formats, and the format of a data center can change anytime without prior note, the "federator" needs to analyze any response first to detect its format of the day, then fully parse !
the response in order to convert to one homogeneous format before merging and returning. Steps which are not required if the responses of all services of the same type and version are (technically) compatible.
Now you can say, with some good reason: let the developers of the "federator" add some money and time and cpu power, and solve that problem, so we don't need to deal with it at the FDSN.
But the world is more than IRIS plus Europe, and not all data centers are federated.
So in all other places where scientists use data from multiple sources, they have to do the same task and check again and again: Do all my data sources still provide the same format? If not: how do I convert/unify? What are the impacts on my subsequent processing?
My fear is that overall, for the entire seismological community, this is more of a hassle than the one of the data centers configuring a new service URL for a service version providing a new response format (a response technically not fulfilling the previous specification).
Kind regards, Philipp
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org <fdsn-wg3-products-unsubscribe<at>lists.fdsn.org>
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
----------------------
FDSN Working Group III
Topic home: http://www.fdsn.org/message-center/topic/fdsn-wg3-products/ | Unsubscribe: fdsn-wg3-products-unsubscribe<at>lists.fdsn.org <fdsn-wg3-products-unsubscribe<at>lists.fdsn.org>
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
-
-
-
Chad Trabant
Philip Crotwell
Robert Casey
Jeremy Fee
Jonathan Schaeffer
Helle PEDERSEN