Discussion:
Proposal to create and maintain acid project per standards revision
(too old to reply)
Rick C. Hodgin
2015-11-19 14:48:04 UTC
Permalink
Raw Message
I propose that a small acid project be created, one which touches on every
aspect of the C standard's particular definition, which allows a person to
see by practical example what each part refers to, and then to have the
standard annotated with the file and line number reference in the acid
project.

Every person would then only need to execute:

mycc acid.c // Should compile and run correctly
mycc acidfail.c // Should generate specific warnings and errors

// Note: There may need to be acidfailN.c versions

The acid project would have all necessary #include files, and features of
that standard revision exposed, so that if mycc is a conforming C compiler
for that revision, it will compile with or without warnings or errors as
appropriate.

Every compiler writer, and every end-user, would then only need to use
that metric / benchmark to determine for themselves if their compiler is
compliant.

As new source code peculiarities are found, they can be added to the
updated acid project, which then reflects the new things identified which
may trip developers up in writing software.

Best regards,
Rick C. Hodgin
m***@yahoo.co.uk
2015-11-19 15:04:45 UTC
Permalink
Raw Message
Post by Rick C. Hodgin
I propose that a small acid project be created,
small?

It seems to me that it would be a *huge* project. It would be very hard indeed. For a start, how would you check that the implementation correctly
implemented the implementation defined things? (If it doesn't, then it's
not a compliant compiler.) That becomes important when you consider
questions like "what can you pass as the first argument of fopen?"

Also, it is almost certain that there are no compliant compilers in
existence ... but the examples of non-compliance (also knows as "bugs")
only show up in certain very specific cases. How would you tell you have
covered all the cases?
Rick C. Hodgin
2015-11-19 15:14:33 UTC
Permalink
Raw Message
Post by m***@yahoo.co.uk
Post by Rick C. Hodgin
I propose that a small acid project be created,
small?
It seems to me that it would be a *huge* project. It would be very hard indeed. For a start, how would you check that the implementation correctly
implemented the implementation defined things? (If it doesn't, then it's
not a compliant compiler.) That becomes important when you consider
questions like "what can you pass as the first argument of fopen?"
Also, it is almost certain that there are no compliant compilers in
existence ... but the examples of non-compliance (also knows as "bugs")
only show up in certain very specific cases. How would you tell you have
covered all the cases?
It would need be to exhaustive. However, it's impractical at this point
to make it exhaustive because the standards document is so big. So, we
would contact major C compiler authors and ask them to contribute ideas
and code on what to include. As more tests are needed by students,
professionals, hobbyists, they can all be passed through a brief review
where their need is validated against currently non-existent examples
which touch that/those aspect(s) of implementation, and then added if
need be.

I'm thinking basic things for now ... signed/unsigned, macro expansion,
standard use including warnings about deprecated declaration syntaxes,
and so on.

Others would know better than I what to include for the host of basic
tests. But I'm thinking a acid.c file that's 1,000 lines long, with a
few small acidN.h #include files to test out that functionality, and a
few acidfailN.c files with acidfailN_N.h #include files, etc.

Best regards,
Rick C. Hodgin
Jakob Bohm
2015-11-19 15:29:00 UTC
Permalink
Raw Message
Post by Rick C. Hodgin
I propose that a small acid project be created, one which touches on every
aspect of the C standard's particular definition, which allows a person to
see by practical example what each part refers to, and then to have the
standard annotated with the file and line number reference in the acid
project.
mycc acid.c // Should compile and run correctly
mycc acidfail.c // Should generate specific warnings and errors
// Note: There may need to be acidfailN.c versions
The acid project would have all necessary #include files, and features of
that standard revision exposed, so that if mycc is a conforming C compiler
for that revision, it will compile with or without warnings or errors as
appropriate.
Note that to test the correct distribution of standard library features
among the standard include files, more than one .c file is needed .
For example it should be tested that if only "<stdio.h>" is included,
then all of the required functions, types and definitions are there
(fopen, printf, FILE*, EOF, ...) and similar for each of the other
standard headers.

I am unsure if there are any cases in standard C (as opposed to vendor
extensions) where some symbol is required to be available only if
certain pairs of include files are used together (as is the case for
POSIX for example).
Post by Rick C. Hodgin
Every compiler writer, and every end-user, would then only need to use
that metric / benchmark to determine for themselves if their compiler is
compliant.
As new source code peculiarities are found, they can be added to the
updated acid project, which then reflects the new things identified which
may trip developers up in writing software.
Depending on licensing issues (which could be difficult), perhaps a lot
of candidate examples could be extracted from the POSIX compliance
tests that implementations must pass to use the UNIX trademark.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
Geoff
2015-11-20 02:33:59 UTC
Permalink
Raw Message
On Thu, 19 Nov 2015 06:48:04 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
I propose that a small acid project be created, one which touches on every
aspect of the C standard's particular definition, which allows a person to
see by practical example what each part refers to, and then to have the
standard annotated with the file and line number reference in the acid
project.
mycc acid.c // Should compile and run correctly
mycc acidfail.c // Should generate specific warnings and errors
// Note: There may need to be acidfailN.c versions
The acid project would have all necessary #include files, and features of
that standard revision exposed, so that if mycc is a conforming C compiler
for that revision, it will compile with or without warnings or errors as
appropriate.
Every compiler writer, and every end-user, would then only need to use
that metric / benchmark to determine for themselves if their compiler is
compliant.
As new source code peculiarities are found, they can be added to the
updated acid project, which then reflects the new things identified which
may trip developers up in writing software.
The template for such a project exists.

It's called "The Standard C Library" by P.J. Plauger. c. 1992
ISBN 0-13-1315099 Prentice Hall


It was written as a reference implementation of the ISO 9899:1990
standard library functions and the test code for those functions.

It's an excellent book.

Anyone attempting to test their current implementation against C90 or
later would be well advised to begin with the functions defined in
that book.
Rick C. Hodgin
2015-11-20 12:35:32 UTC
Permalink
Raw Message
Post by Geoff
The template for such a project exists.
It's called "The Standard C Library" by P.J.
Plauger. c. 1992 ISBN 0-13-1315099 Prentice Hall
A good start.

Best regards,
Rick C. Hodgin
Geoff
2015-11-20 18:15:06 UTC
Permalink
Raw Message
On Fri, 20 Nov 2015 04:35:32 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
Post by Geoff
The template for such a project exists.
It's called "The Standard C Library" by P.J.
Plauger. c. 1992 ISBN 0-13-1315099 Prentice Hall
A good start.
Every compiler that claims to be ISO C compliant (even partially, to
whatever version) should be shipped with the source code of the
compliance test suites included. They had to do the work to prove to
themselves it was compliant, why should the end users have to design
their own tests? Or are they supposed to take it on faith that the
work was done? Proprietary? Well, where's the ISO reference
implementation or test suite?
Rick C. Hodgin
2015-11-20 18:25:37 UTC
Permalink
Raw Message
Post by Geoff
On Fri, 20 Nov 2015 04:35:32 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
Post by Geoff
The template for such a project exists.
It's called "The Standard C Library" by P.J.
Plauger. c. 1992 ISBN 0-13-1315099 Prentice Hall
A good start.
Every compiler that claims to be ISO C compliant (even partially, to
whatever version) should be shipped with the source code of the
compliance test suites included. They had to do the work to prove to
themselves it was compliant, why should the end users have to design
their own tests? Or are they supposed to take it on faith that the
work was done? Proprietary? Well, where's the ISO reference
implementation or test suite?
I think the C Standards committee should maintain the various versions
for independent download. Compiler authors could include a copy of the
standard test per their release date as well, but it would be too easy
and arguably tempting for some of them to reach into the source code
and provide a tweak or two which changes the test to give their compiler
a better result.

I think the acid test is a requirement for the C programming language.
I'm amazed that one doesn't already exist, and isn't maintained. It
would probably make it easier for compiler developers to introduce more
support for standards compliance as they have real coding examples to
use, rather than reading through words describing the thing.

Best regards,
Rick C. Hodgin
Geoff
2015-11-20 19:08:05 UTC
Permalink
Raw Message
On Fri, 20 Nov 2015 10:25:37 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
Post by Geoff
On Fri, 20 Nov 2015 04:35:32 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
Post by Geoff
The template for such a project exists.
It's called "The Standard C Library" by P.J.
Plauger. c. 1992 ISBN 0-13-1315099 Prentice Hall
A good start.
Every compiler that claims to be ISO C compliant (even partially, to
whatever version) should be shipped with the source code of the
compliance test suites included. They had to do the work to prove to
themselves it was compliant, why should the end users have to design
their own tests? Or are they supposed to take it on faith that the
work was done? Proprietary? Well, where's the ISO reference
implementation or test suite?
I think the C Standards committee should maintain the various versions
for independent download. Compiler authors could include a copy of the
standard test per their release date as well, but it would be too easy
and arguably tempting for some of them to reach into the source code
and provide a tweak or two which changes the test to give their compiler
a better result.
I think the acid test is a requirement for the C programming language.
I'm amazed that one doesn't already exist, and isn't maintained. It
would probably make it easier for compiler developers to introduce more
support for standards compliance as they have real coding examples to
use, rather than reading through words describing the thing.
I had three cross-compilers for some embedded systems I was developing
back in 1992. They all claimed a certain degree of compliance with the
standard of the time. I wrote code for the system and occasionally
found several anomalies in the compiler and in the generated code. I
spent several hours verifying and documenting the bugs and reporting
them to the vendors whereupon they said, "Oh, yeah, you're right. Let
me fix that and send you an update." Those were fun times. :)

I am entirely in favor of your acid test. Now, get to work. :)
Rick C. Hodgin
2015-11-20 19:11:25 UTC
Permalink
Raw Message
Post by Geoff
On Fri, 20 Nov 2015 10:25:37 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
Post by Geoff
On Fri, 20 Nov 2015 04:35:32 -0800 (PST), "Rick C. Hodgin"
Post by Rick C. Hodgin
Post by Geoff
The template for such a project exists.
It's called "The Standard C Library" by P.J.
Plauger. c. 1992 ISBN 0-13-1315099 Prentice Hall
A good start.
Every compiler that claims to be ISO C compliant (even partially, to
whatever version) should be shipped with the source code of the
compliance test suites included. They had to do the work to prove to
themselves it was compliant, why should the end users have to design
their own tests? Or are they supposed to take it on faith that the
work was done? Proprietary? Well, where's the ISO reference
implementation or test suite?
I think the C Standards committee should maintain the various versions
for independent download. Compiler authors could include a copy of the
standard test per their release date as well, but it would be too easy
and arguably tempting for some of them to reach into the source code
and provide a tweak or two which changes the test to give their compiler
a better result.
I think the acid test is a requirement for the C programming language.
I'm amazed that one doesn't already exist, and isn't maintained. It
would probably make it easier for compiler developers to introduce more
support for standards compliance as they have real coding examples to
use, rather than reading through words describing the thing.
I had three cross-compilers for some embedded systems I was developing
back in 1992. They all claimed a certain degree of compliance with the
standard of the time. I wrote code for the system and occasionally
found several anomalies in the compiler and in the generated code. I
spent several hours verifying and documenting the bugs and reporting
them to the vendors whereupon they said, "Oh, yeah, you're right. Let
me fix that and send you an update." Those were fun times. :)
I am entirely in favor of your acid test. Now, get to work. :)
I am currently working on a close off-shoot of the C language, and plan
to include something like this for my compiler. I ultimately also plan
to provide C90 and C99 support, so I will have some type of acid tests
for those at least (Lord willing).

Best regards,
Rick C. Hodgin
a***@gmail.com
2015-12-08 09:08:53 UTC
Permalink
Raw Message
I am interested in an open source C validation suite. I am the author of https://github.com/andrewchambers/c. You can see my ad hoc solution in the test directory.
Loading...