Discussion:
[clisp-list] bug: :if-exists :append
Jean Louis
2017-02-13 04:45:29 UTC
Permalink
Hello,

I cannot file bug on Sourceforge for some reason, so maybe someone
else may do it. I hope this bug will be corrected at least in the
development version.

When file is chattr-ed +a (append only) on GNU/Linux system, CLISP is
reporting: *** - OS-FILE-ERROR(EPERM): Operation not permitted
and is not appending to the file.

To reproduce, touch my-append-only-file and:
sudo chattr +a my-append-only-file

and try:

(with-open-file
(str "my-append-only-file" :direction :output :if-exists :append :if-does-not-exist :error)
(format str "~a~%" "Hello"))


Jean
Sam Steingold
2017-02-13 18:51:10 UTC
Permalink
Hi Jean,
Post by Jean Louis
I cannot file bug on Sourceforge for some reason, so maybe someone
else may do it. I hope this bug will be corrected at least in the
development version.
When I click on "Create Ticket" on https://sourceforge.net/p/clisp/bugs/,
I am redirected to https://sourceforge.net/p/clisp/bugs/new/
What happens to you?
Post by Jean Louis
When file is chattr-ed +a (append only) on GNU/Linux system, CLISP is
reporting: *** - OS-FILE-ERROR(EPERM): Operation not permitted
and is not appending to the file.
To be honest, I am not surprised.
CLISP was written before chattr came around, so, with `:if-exists
:append` it opens the file the usual way and fseek()s to the end.

Moreover, a comment in clisp/src/stream.d says:

--8<---------------cut here---------------start------------->8---
/* treat Mode :APPEND:
CLHS says that :APPEND implies that "the file pointer is _initially_
positioned at the end of the file". Note that this is different from
the Unix O_APPEND semantics. */
--8<---------------cut here---------------end--------------->8---

for the reference (http://pubs.opengroup.org/onlinepubs/7908799/xsh/open.html):
--8<---------------cut here---------------start------------->8---
O_APPEND
If set, the file offset will be set to the end of the file prior to each write.
--8<---------------cut here---------------end--------------->8---

IOW, to open your `chattr +a` file, we need to pass `O_APPEND` to `open(2)`.
However, this will break this code:

--8<---------------cut here---------------start------------->8---
(with-open-file (s "my-append-only-file" :direction :output :if-exists :append)
(let ((pos (file-position s)))
(write-string "foo" s)
(file-position s (- pos 3))
(write-string "bar" s)))
--8<---------------cut here---------------end--------------->8---

ANSI CL mandates that this appends 3 chars "bar" to
"my-append-only-file".
However, if "my-append-only-file" was opened with `O_APPEND`, it will
actually append 6 chars "foobar".

It is not clear to me what to do here:

* `file-position' can detect that the file is O_APPEND and return nil -
this is a functionality loss in many ways - e.g., we cannot scroll
back and read an O_RDWR|O_APPEND file.

* Accept that the code above will not do what the user expects for
___SOME___ streams.

* Add another `:direction' option -- `:unix-append' which will use
`O_APPEND'.

I think the best course of action if the last one - it will be backwards
compatible and will not violate too many expectations.

For now the only course of action is to use FFI or syscall (not sure if
it can do that) to get a raw file descriptor and then wrap it into a
Lisp stream using [`make-stream`](http://clisp.org/impnotes/make-stream.html).
Post by Jean Louis
sudo chattr +a my-append-only-file
(with-open-file
(str "my-append-only-file" :direction :output :if-exists :append :if-does-not-exist :error)
(format str "~a~%" "Hello"))
--
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://think-israel.org
http://www.memritv.org http://jij.org http://no2bds.org
Those who can't write, write manuals.
Jean Louis
2017-02-13 19:20:58 UTC
Permalink
Hello,

Thank you, I understand it.
Post by Sam Steingold
When I click on "Create Ticket" on https://sourceforge.net/p/clisp/bugs/,
I am redirected to https://sourceforge.net/p/clisp/bugs/new/
What happens to you?
I guess I don't have account there. I have account only on mailing
list. Not that I am fond of sourceforge especially for GNU projects.

OK, if you happen to know how to get file descriptor by sys:: let me
know. Or how to append to file which is chattr +a

I have tried that with SBCL and it appended to the file, the technical
stuff behind it is well explained from your side, on my side, I can
just observe what happened, cannot observe it. Now I am using shell
echo ~a >> to the file.

Jean
Post by Sam Steingold
IOW, to open your `chattr +a` file, we need to pass `O_APPEND` to `open(2)`.
--8<---------------cut here---------------start------------->8---
(with-open-file (s "my-append-only-file" :direction :output :if-exists :append)
(let ((pos (file-position s)))
(write-string "foo" s)
(file-position s (- pos 3))
(write-string "bar" s)))
--8<---------------cut here---------------end--------------->8---
ANSI CL mandates that this appends 3 chars "bar" to
"my-append-only-file".
However, if "my-append-only-file" was opened with `O_APPEND`, it will
actually append 6 chars "foobar".
* `file-position' can detect that the file is O_APPEND and return nil -
this is a functionality loss in many ways - e.g., we cannot scroll
back and read an O_RDWR|O_APPEND file.
* Accept that the code above will not do what the user expects for
___SOME___ streams.
* Add another `:direction' option -- `:unix-append' which will use
`O_APPEND'.
I think the best course of action if the last one - it will be backwards
compatible and will not violate too many expectations.
For now the only course of action is to use FFI or syscall (not sure if
it can do that) to get a raw file descriptor and then wrap it into a
Lisp stream using [`make-stream`](http://clisp.org/impnotes/make-stream.html).
Sam Steingold
2017-02-13 21:18:12 UTC
Permalink
Post by Jean Louis
Post by Sam Steingold
When I click on "Create Ticket" on https://sourceforge.net/p/clisp/bugs/,
I am redirected to https://sourceforge.net/p/clisp/bugs/new/
What happens to you?
I guess I don't have account there. I have account only on mailing
list. Not that I am fond of sourceforge especially for GNU projects.
Indeed, you need an account.
Please do get it.
We appreciate your bug reports.
Post by Jean Louis
OK, if you happen to know how to get file descriptor by sys:: let me
know. Or how to append to file which is chattr +a
(require "linux")
(defparameter *my-stream*
(ext:make-stream (linux:open filename (logior linux::O_APPEND linux::O_RDONLY) 0)
:direction :append ...))
Post by Jean Louis
I have tried that with SBCL and it appended to the file
interesting - what does my lisp code below do in SBCL?
Post by Jean Louis
Post by Sam Steingold
--8<---------------cut here---------------start------------->8---
(with-open-file (s "my-append-only-file" :direction :output :if-exists :append)
(let ((pos (file-position s)))
(write-string "foo" s)
(file-position s (- pos 3))
(write-string "bar" s)))
--8<---------------cut here---------------end--------------->8---
ANSI CL mandates that this appends 3 chars "bar" to
"my-append-only-file".
However, if "my-append-only-file" was opened with `O_APPEND`, it will
actually append 6 chars "foobar".
--
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net https://ffii.org
https://jihadwatch.org http://honestreporting.com http://www.dhimmitude.org
Selling grief is easier than buying happiness.
Sam Steingold
2017-02-14 14:32:43 UTC
Permalink
Jean,
please keep the discussion on the list.
Post by Sam Steingold
Post by Jean Louis
OK, if you happen to know how to get file descriptor by sys:: let me
know. Or how to append to file which is chattr +a
(require "linux")
(defparameter *my-stream*
(ext:make-stream (linux:open filename (logior linux::O_APPEND linux::O_RDONLY) 0)
:direction :append ...))
I cannot require "linux" for some reason, using 2.49+ development
version which is too hard to compile.
this the "bindings/glibc" module.
http://clisp.org/impnotes/modules.html#incmod-bind
Post by Sam Steingold
Post by Jean Louis
I have tried that with SBCL and it appended to the file
interesting - what does my lisp code below do in SBCL?
Post by Jean Louis
Post by Sam Steingold
--8<---------------cut here---------------start------------->8---
(with-open-file (s "my-append-only-file" :direction :output :if-exists :append)
(let ((pos (file-position s)))
(write-string "foo" s)
(file-position s (- pos 3))
(write-string "bar" s)))
--8<---------------cut here---------------end--------------->8---
debugger invoked on a TYPE-ERROR in thread
The value
-3
is not of type
(OR UNSIGNED-BYTE (MEMBER NIL :END :START))
oops, my bad.

--8<---------------cut here---------------start------------->8---
(with-open-file (s "my-append-only-file" :direction :output :if-exists :append)
(let ((pos (file-position s)))
(print pos)
(write-string "foo" s)
(print (file-position s))
(file-position s pos)
(print (file-position s))
(write-string "bar" s)
(print (file-position s))))
--8<---------------cut here---------------end--------------->8---

this should work without errors - if it does not, please try `:direction :io`.
the question is: will this append

* "foobar" (as per `O_APPEND`) or
* "bar" (as per `:if-exists :append`)

we should be able to figure that out by examining the printed
file-positions.

Thanks.
--
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://mideasttruth.com
https://ffii.org http://www.memritv.org http://camera.org http://jij.org
"Sustainability" without "population growth control" is hypocrisy.
Jean Louis
2017-02-14 17:21:01 UTC
Permalink
Post by Sam Steingold
Post by Jean Louis
Post by Sam Steingold
When I click on "Create Ticket" on https://sourceforge.net/p/clisp/bugs/,
I am redirected to https://sourceforge.net/p/clisp/bugs/new/
What happens to you?
I guess I don't have account there. I have account only on mailing
list. Not that I am fond of sourceforge especially for GNU projects.
Indeed, you need an account.
Please do get it.
We appreciate your bug reports.
OK, will do.
Post by Sam Steingold
Post by Jean Louis
OK, if you happen to know how to get file descriptor by sys:: let me
know. Or how to append to file which is chattr +a
(require "linux")
(defparameter *my-stream*
(ext:make-stream (linux:open filename (logior linux::O_APPEND linux::O_RDONLY) 0)
:direction :append ...))
I cannot require "linux" for some reason, using 2.49+ development
version which is too hard to compile.
Post by Sam Steingold
Post by Jean Louis
I have tried that with SBCL and it appended to the file
interesting - what does my lisp code below do in SBCL?
Post by Jean Louis
Post by Sam Steingold
--8<---------------cut here---------------start------------->8---
(with-open-file (s "my-append-only-file" :direction :output :if-exists :append)
(let ((pos (file-position s)))
(write-string "foo" s)
(file-position s (- pos 3))
(write-string "bar" s)))
--8<---------------cut here---------------end--------------->8---
debugger invoked on a TYPE-ERROR in thread
#<THREAD "main thread" RUNNING {10028DE633}>:
The value
-3
is not of type
(OR UNSIGNED-BYTE (MEMBER NIL :END :START))


anyway, I cannot test it. For me, I just know, I don't get result.

Jean

----- End forwarded message -----

Loading...