[clisp-list] How to simple exit in CGI?

[Jean Louis]

Previous message was not well formulated:

The "program" is here:

(defun main ()
(ext:exit 0))

is saved as memory image test.cgi

I would like to simply exit as CGI program. When I make the memory
image, and run
http://localhost/test.cgi?a or if I put any char after cgi, like / or ?a=1 the memory image is exiting or giving its output

If I just call it with
wget -O /dev/shm/test "http://localhost/order.cgi"

I get after longer waiting of about 45 seconds following output:

*** - Ctrl-C: User break
Break 1 [1]>


How can I avoid that? It is for the security of the CGI execution, tht
without parameters it should simply exit, but exiting seem a problem to me.

Jean

[Pascal Bourguignon]

Post by Jean Louis
(defun main ()
(ext:exit 0))
is saved as memory image test.cgi
I would like to simply exit as CGI program. When I make the memory
image, and run
http://localhost/test.cgi?a or if I put any char after cgi, like / or ?a=1 the memory image is exiting or giving its output
If I just call it with
wget -O /dev/shm/test "http://localhost/order.cgi"
*** - Ctrl-C: User break
Break 1 [1]>
How can I avoid that? It is for the security of the CGI execution, tht
without parameters it should simply exit, but exiting seem a problem to me.

You should print out HTTP headers. You can use them to report some status.

Something like:

#!/usr/local/bin/clisp -q -ansi -norc -E utf-8
;; -*- mode:lisp;coding:utf-8 -*-

(defun main ()
(write-string "Content-Type: text/plain
Content-Length: 0
Status: 204


")
(finish-output)
(ext:exit 0))

(main)

[Jean Louis]

Post by Pascal Bourguignon

Post by Jean Louis
without parameters it should simply exit, but exiting seem a problem to me.

You should print out HTTP headers. You can use them to report some status.
#!/usr/local/bin/clisp -q -ansi -norc -E utf-8
;; -*- mode:lisp;coding:utf-8 -*-
(defun main ()
(write-string "Content-Type: text/plain
Content-Length: 0
Status: 204
")
(finish-output)
(ext:exit 0))
(main)

Thank you. Sure, I am sending headers when necessary. If not
necessary, in case of direct call to CGI, I like to send the redirect.

I am waiting on either QUERY_STRING for GET parameters or
*standard-input* for parameters from POST.

However, if there is no QUERY_STRING and no standard input sent, then
I get in trouble.

If I am to use it as script, at least it aborts, and gives me nothing:

wget -O /dev/shm/test "http://localhost/test.cgi"; less /dev/shm/test
--2017-03-28 17:21:18-- http://localhost/test.cgi
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified
Saving to: ‘/dev/shm/test’

/dev/shm/test [ <=> ] 0 --.-KB/s in 0s

2017-03-28 17:21:18 (0.00 B/s) - ‘/dev/shm/test’ saved [0]

and that is "almost" wanted. Yet the execution does not arrive to the
redirect. So the page does not get redirected. Why?

#!/usr/local/bin/clisp -q -ansi -norc -E utf-8
(setf custom:*ansi* t)
(defun main nil
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(alexandria.0.dev:read-stream-content-into-string *standard-input*))))
(format t "Status: 302 Found~%Location: ~a~%~%" url)
(finish-output)
(exit 0)))
(main)


If I then save it as memory image, exactly this one here below, and
then do following, I get redirect: wget -O /dev/shm/test
"http://localhost/test.cgi?a";

yet if I don't send any parameters, I get displayed:

*** - Ctrl-C: User break
Break 1 [1]>

I would like to understand what is happening.

Jean

[J***@t-systems.com]

Hi,

Post by Pascal Bourguignon
You should print out HTTP headers. You can use them to report some status.
Thank you. Sure, I am sending headers when necessary. If not necessary,
in case of direct call to CGI, I like to send the redirect.

Please explain "in case of direct call to CGI"?
IMHO, you should learn the HTTP protocol, either version 1.0 (1996)
https://tools.ietf.org/html/rfc1945
or 1.1 (1999) or perhaps HTTP/2 (2015).
There's no "when necessary". You MUST always return headers. It is ALWAYS necessary.
(Except with HTTP "0.9", 26 years ago).

Post by Pascal Bourguignon
(query (or (getenv "QUERY_STRING")

Please explain your setup. Obviously CLISP is not the web-server front end
(otherwise there would be no such environment variables).
What is your web-server (Apache?)
and how is it configured to run CLISP (mod_fcgi for FastCGI)?

You need to read the documentation about how your web-server front end
invokes the CGI executables. Then you will know what to expect from the command line,
environment variables and/or standard input.

Post by Pascal Bourguignon
*** - Ctrl-C: User break
Break 1 [1]>
I would like to understand what is happening.

Me too, yet your configuration is completely unclear to me. When I ran CLISP behind
Apache many years ago, I'd never see a ^C in any Apache log, nor a command prompt.
How is CLISP started in your setup?

If you want one CLISP memory image to be both useable from the command line
and when invoked from Apache as a cgi-bin, you need to take different path
of execution, after detecting whether the program was called in a CGI environment.

Regards,
Jörg Höhle

[Jean Louis]

Post by J***@t-systems.com

Post by Pascal Bourguignon
You should print out HTTP headers. You can use them to report some status.
Thank you. Sure, I am sending headers when necessary. If not necessary,
in case of direct call to CGI, I like to send the redirect.

Please explain "in case of direct call to CGI"?
IMHO, you should learn the HTTP protocol, either version 1.0 (1996)
https://tools.ietf.org/html/rfc1945
or 1.1 (1999) or perhaps HTTP/2 (2015).
There's no "when necessary". You MUST always return headers. It is ALWAYS necessary.
(Except with HTTP "0.9", 26 years ago).

Alright, so I wish to print the redirect in case of no input. Yet, it
never comes to the redirect line.

Post by J***@t-systems.com

Post by Pascal Bourguignon
(query (or (getenv "QUERY_STRING")

Please explain your setup. Obviously CLISP is not the web-server front end
(otherwise there would be no such environment variables).
What is your web-server (Apache?)

thttpd

yet it does not matter, I tried with other web servers, it is no any
fault in the web server.

wget -O /dev/shm/test "http://localhost/test.cgi"; less /dev/shm/test
--2017-03-28 19:33:48-- http://localhost/test.cgi
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response...

that line is waiting for about 45 seconds and answer with nothing. The
program is down below, and it is saved as executable memory image.

If I run it with:

wget -O /dev/shm/test "http://localhost/test.cgi?anything"; less
/dev/shm/test

then I immediately get the redirect to other page.

(setf custom:*ansi* t)
(defun main nil
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(alexandria.0.dev:read-stream-content-into-string *standard-input*))))
(format t "Status: 302 Found~%Location: ~a~%~%" url) ;; XXX line
(finish-output)
(exit 0)))
;; (main)

Post by J***@t-systems.com
and how is it configured to run CLISP (mod_fcgi for FastCGI)?

It is just running the executable memory image test.cgi

Post by J***@t-systems.com

Post by Pascal Bourguignon
*** - Ctrl-C: User break
Break 1 [1]>
I would like to understand what is happening.

Me too, yet your configuration is completely unclear to me. When I
ran CLISP behind Apache many years ago, I'd never see a ^C in any
Apache log, nor a command prompt. How is CLISP started in your
setup?

Just as you see the above small program, that is the one, and it is
saved as executable memory image test.cgi -- and I am using CLISP
development version.

If I am not sending anything, the XXX line above never executes, so
CLISP is waiting and get Ctrl-C probably by web server who interrupts
the connection.

Thank you. If you know the anwer what I am doing wrong that the line
with (format t "Status 302... never comes to execution, without
parameters, let me know.

Jean

[Jean Louis]

This is how the output looks like in browser:
Loading Image...

From this snippet below, when saved as executable memory image with

init function 'main, and when called without arguments (POST/GET).

(setf custom:*ansi* t)
(defun main nil
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(alexandria.0.dev:read-stream-content-into-string *standard-input*))))
(format t "Status: 302 Found~%Location: ~a~%~%" url)
(finish-output)
(exit 0)))
;; (main)


Jean

[Pascal Bourguignon]

Post by Jean Louis
Thank you. Sure, I am sending headers when necessary. If not
necessary, in case of direct call to CGI, I like to send the redirect.
I am waiting on either QUERY_STRING for GET parameters or
*standard-input* for parameters from POST.
However, if there is no QUERY_STRING and no standard input sent, then
I get in trouble.
wget -O /dev/shm/test "http://localhost/test.cgi"; less /dev/shm/test
--2017-03-28 17:21:18-- http://localhost/test.cgi
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified
Saving to: ‘/dev/shm/test’
/dev/shm/test [ <=> ] 0 --.-KB/s in 0s
2017-03-28 17:21:18 (0.00 B/s) - ‘/dev/shm/test’ saved [0]
and that is "almost" wanted. Yet the execution does not arrive to the
redirect. So the page does not get redirected. Why?
#!/usr/local/bin/clisp -q -ansi -norc -E utf-8
(setf custom:*ansi* t)
(defun main nil
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(alexandria.0.dev:read-stream-content-into-string *standard-input*))))
(format t "Status: 302 Found~%Location: ~a~%~%" url)
(finish-output)
(exit 0)))
(main)
If I then save it as memory image, exactly this one here below, and
then do following, I get redirect: wget -O /dev/shm/test
"http://localhost/test.cgi?a";
*** - Ctrl-C: User break
Break 1 [1]>
I would like to understand what is happening.

There is some confusion between the type of delivery you want, and in the case of CGI, some technical difficulty in writing a file able to do both.


When you write a file starting with “#!” it’s to be considered as a script by the unix system, that is, a file that will be processed (interpreted) by the command given after “#!”.
Read: http://www.clisp.org/impnotes/quickstart.html#quickstart-unix <http://www.clisp.org/impnotes/quickstart.html#quickstart-unix>

Post by Jean Louis
#!/usr/local/bin/clisp -q -ansi -norc -E utf-8

works. Notice that -ansi is equivalent to (setf custom:*ansi* t) so as a script, you should not need this line. (However if you consider loading the file in a clisp image, then the shebang line will be ignored (there’s a clisp specific dispatching reader macro on #\# #\! to treat this line as a comment).

If you save the above file in test.cgi and chmod +x it, it won’t work, because alexandria won’t be loaded by the script.

You could load ~/quicklisp/setup.lisp and then use quicklisp to load alexandria:

#!/usr/local/bin/clisp -q -ansi -norc -E utf-8
;; -*- mode:lisp;coding:utf-8 -*-
(in-package "COMMON-LISP-USER")

(load #P"~/quicklisp/setup.lisp")
(ql:quickload :alexandria)

(defun main (&optional (arguments *args*))
(declare (ignore arguments))
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(alexandria:read-stream-content-into-string *standard-input*))))
(format t "Status: 302 Found~%Location: ~a~%~%" url)
(finish-output)
(exit 0)))

(main *args*)

but this won’t work either, in the case of a CGI, because quickload is quite verbose, and will print a lot of messages before your CGI script starts executing. You can see it by running the script directly in the shell (this is the big advantage of CGI, that you can run and test them in the shell, without going thru HTTP).


You could use a macro such as without-output or redirecting-stdout-to-stderr to prevent quicklisp messages:

(defmacro redirecting-stdout-to-stderr (&body body)
(let ((verror (gensym))
(voutput (gensym)))
`(let* ((,verror nil)
(,voutput (with-output-to-string (stream)
(let ((*standard-output* stream)
(*error-output* stream)
(*trace-output* stream))
(handler-case (progn ,@body)
(error (err) (setf ,verror err)))))))
(when ,verror
(terpri *error-output*)
(princ ,voutput *error-output*)
(terpri *error-output*)
(princ ,verror *error-output*)
(terpri *error-output*)
(terpri *error-output*)
#-testing-script (exit ex-software)))))

(defun getpid ()
(or (ignore-errors (find-symbol "getpid" "LINUX"))
(ignore-errors (find-symbol "PROCESS-ID" "OS"))
(ignore-errors (find-symbol "PROCESS-ID" "SYSTEM"))))


(defun report-the-error (err string-stream)
(let ((log-path (format nil "/tmp/~A.~D.errors" *program-name*
(let ((getpid (getpid)))
(if getpid
(funcall getpid)
"nopid")))))
(with-open-file (log-stream log-path
:direction :output
:if-exists :supersede
:if-does-not-exist :create)
(format log-stream "~A GOT AN ERROR: ~A~%~80,,,'-<~>~%~A~%"
*program-name* err (get-output-stream-string string-stream)))
(format *error-output* "~A: ~A~% See ~A~%" *program-name* err log-path)
(finish-output *error-output*)
(exit ex-software)))

(defmacro without-output (&body body)
`(prog1 (values)
(with-output-to-string (net)
(handler-case
(let ((*standard-output* net)
(*error-output* net)
(*trace-output* net))
,@body)
(error (err) (report-the-error err net))))))


(without-output (ql:quickload :alexandria))

By the way, you should normally use the nickname ALEXANDRIA, instead of the specific versioned package name ALEXANDRIA.0.DEV, unless you are using an internal API that you can expect to be specific to the version 0 (development) of ALEXANDRIA.


Now, if you don’t plan to use scripts for your CGI, but an executable image, I would advise you to structure your project with:

- a normal lisp source file,
- a lisp source file defining a package,
- an ASDF system definition file, to load your program with asdf or quicklisp,
- a lisp “script” to load and generate the executable image, and
- a Makefile to build it.
See attached files.

CL-USER> (ls)
./Makefile
./generate-executable.lisp
./jean-cgi.asd
./jean-cgi.lisp
./packages.lisp
; No value
CL-USER> (ql:quickload "jean-cgi")
To load "jean-cgi":
Load 1 ASDF system:
jean-cgi
; Loading "jean-cgi"

("jean-cgi")
CL-USER>

The advantage of being able to load your program in an interactive lisp image, is that you will be able to run and test it at the REPL, with the help of the interactive debugger, etc.
For example, this should work:

CL-USER> (with-input-from-string (*standard-intput* "")
(jean-cgi:main))

Well, unsurprisingly, it blocks in alexandria. I often find that alexandria does a bad job.
Here, it seems that it is able to read an empty *standard-input* stream when it’s connected to the terminal or an empty file, but not when it’s a string-stream.

My own utility library cesarum, doesn’t have this problem:

CL-USER> (with-input-from-string (*standard-intput* "")
(contents-from-stream *standard-intput* :min-size 16384))
""

you might prefer using it rather than alexandria (but beware, I license my software under the AGPL3 license). cesarum currently can be found at: https://gitlab.com/com-informatimago/com-informatimago <https://gitlab.com/com-informatimago/com-informatimago> ; clone it in ~/quicklisp/local-projects and use (ql:quickload :com.informatimago.common-lisp.cesarum) to load it.


Anyways, once you’ve debugged your program in an interactive REPL, it’s time to generate the executable image. Here, one objective may be to be able to do it in a reproducible way: you need to control the environment so that each time you or somebody else build it, you all build the same thing. In particular, you don’t want to inherit the environment loaded from the rc file, or found in an interactive REPL. For this reason, I use a generate-executable.lisp file, which is loaded without rc files, and that will set up the environment saved in the lisp image as precisely and reproductibly (notably across various CL implementations) as possible.

And to help, I write a Makefile that will invoke the various CL implementations with the precise options to load generate-executable.lisp correctly (it changes from one implementation to another).



[***@despina :0.0]$ make
clisp -q -ansi -norc -x '(load "generate-executable.lisp")' -x '(quit)'
;; Loading file generate-executable.lisp ...
;; Loading file /Users/pjb/quicklisp/setup.lisp ...
;; Loading file /Users/pjb/quicklisp/ASDF.lisp ...
;; Loaded file /Users/pjb/quicklisp/ASDF.lisp
;; Loaded file /Users/pjb/quicklisp/setup.lisp
To load "jean-cgi":
Load 1 ASDF system:
jean-cgi
; Loading "jean-cgi"


Bytes permanently allocated: 172,224
Bytes currently in use: 6,034,936
Bytes available until next GC: 1,507,604
;; Loaded file generate-executable.lisp
T

~/Sites/cgi
[***@despina :0.0]$

So you can see how clisp is invoked to load generate-executable.lisp.
generate-executable.lisp loads quicklisp, and then use it to compile
and load jean-cgi. Finally it saves the executable image.

In more complex programs, generate-executable.lisp would first delete the compiled fas files found in ~/.cache/common-lisp/*/… to ensure that all the fas file loaded by quicklisp are freshly compiled (notably with the *features* set that can be configured in generate-executable.lisp).

Finally, we’ve obtained a CGI executable that we can test in the shell:

[***@despina :0.0]$ ./jean.cgi </dev/null
Status: 302 Found
Location: http://localhost


~/Sites/cgi
[***@despina :0.0]$

Then invoking it from HTTP is a mere formality.

[***@despina :0.0]$ curl --head 'http://localhost/~pjb/cgi/jean.cgi'
HTTP/1.1 302 Found
Date: Tue, 28 Mar 2017 19:41:14 GMT
Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 OpenSSL/1.0.2k DAV/2
Location: http://localhost
Content-Type: text/plain


~/Sites/cgi
[***@despina :0.0]$


Have fun!

[Jean Louis]

Post by Pascal Bourguignon
There is some confusion between the type of delivery you want, and
in the case of CGI, some technical difficulty in writing a file able
to do both.

I am just trying to understand, if there is no *standard-input*
coming, why the program is not going one step forward, to for example,
spit the headers our.

Post by Pascal Bourguignon
When you write a file starting with “#!” it’s to be considered as a
script by the unix system, that is, a file that will be processed
(interpreted) by the command given after “#!”.

Thank you for explanation, I am aware of it. Yet I am trying to run it
as a memory image, for the speed it is giving.

Post by Pascal Bourguignon
If you save the above file in test.cgi and chmod +x it, it won’t
work, because alexandria won’t be loaded by the script.

Right now, it works, as it is saved as memory image which itself has
Alexandria inside. I am using this small program to detect the change
when test.lisp is saved, so that it generates test.cgi which I can
then try to wget or browse.

(defparameter file (car *args*))

(defun check-the (file)
(let ((file (format nil "~a.lisp" file)))
(file-stat-mtime (file-stat file))))

(defparameter check (check-the file))

(defun do-compile (file) ;; (setf file "order")
(cd "/home/data1/protected/public_html/")
(let* ((original (format nil "~a.lisp" file))
(executable (format nil "~a.cgi" file))
(command (format nil "lisp -i ~a -x \"(saveinitmem \\\"~a\\\" :executable t :init-function 'main :quiet t :norc t)\"" original executable)))
(unless (equalp check (check-the file))
(print command)
(shell command)
(set-file-stat executable :mode #O0755)
(setf check (check-the file))))
(sleep .3))

(loop (do-compile file))

I can use it without Alexandria, I guess this is equivalent
below. Yet, I just need to find out how to recognize that there is no
*standard-input* coming, as it is more or less "hanging" if there is
no POST or GET parameter.

(setf custom:*ansi* t)

(defun POST-collect ()
(let (lines (loop for line = (read-line *standard-input* nil nil)
while line
collect))
(format "~{~a~}" lines)))

(defun headers nil
(format t "Content-type: text/plain~2%"))

(defun main ()
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(POST-collect))))
;; (format t "Status: 302 Found~%Location: ~a~%~%" url)
(headers)
(princ "Hello there")
;; (finish-output)
(exit 0)))
;; (main)

Post by Pascal Bourguignon
You could use a macro such as without-output or

Well thank you for that, I will keep it for later usage.

I was thinking that :verbose nil and :silent t within (ql:quickload ""
would suppress the messages. And in some scripts it does it so.

So it is possible to say:

(ql:quickload "alexandria" :verbose nil :silent t)
without the output.

Post by Pascal Bourguignon
By the way, you should normally use the nickname ALEXANDRIA, instead
of the specific versioned package name ALEXANDRIA.0.DEV, unless you
are using an internal API that you can expect to be specific to the
version 0 (development) of ALEXANDRIA.

That I did not know, for one reason, when I type alexandria on CLISP
REPL, it completes to alexandria.0.dev: and I was thinking that is
so. Now I see that I can use alexandria: only, thank you.

Post by Pascal Bourguignon
- a normal lisp source file,
- a lisp source file defining a package,
- an ASDF system definition file, to load your program with asdf or quicklisp,
- a lisp “script” to load and generate the executable image, and
- a Makefile to build it.
See attached files.
CL-USER> (ls)
./Makefile
./generate-executable.lisp
./jean-cgi.asd
./jean-cgi.lisp
./packages.lisp
; No value
CL-USER> (ql:quickload "jean-cgi")
jean-cgi
; Loading "jean-cgi"
("jean-cgi")
CL-USER>
The advantage of being able to load your program in an interactive
lisp image, is that you will be able to run and test it at the REPL,
with the help of the interactive debugger, etc. For example, this
CL-USER> (with-input-from-string (*standard-intput* "")
(jean-cgi:main))

Thank you, I am keeping that for later. Right now I have put
everything in one Lisp image, one order.lisp file, including the GnuPG
key, so that GnuPg directory is recreated on server automatically, and
that all inquiries are encrypted on the way back to me.

Everything is working well so far, only that I cannot solve the
problem when the CGI is called directly, it is waiting too long.

If I make the same thing in Perl:

#!/usr/bin/perl
exit;

there is no waiting time. And I came from Perl world.

and if I make the same in CLISP:

(defun main ()
(exit))

and save as memory image, the wget is not exiting. Basically CLISP is
not exiting.

Post by Pascal Bourguignon
CL-USER> (with-input-from-string (*standard-intput* "")
(contents-from-stream *standard-intput* :min-size 16384))
""
you might prefer using it rather than alexandria (but beware, I
license my software under the AGPL3 license).

That is best license currently for free software. I would use AGPL
myself, there is no incompatibility.

OK I have all your repository on my computer.

There is slight problem, there, if I come to directory, common-lisp,
and type make, thee command "lc" is missing. I don't have any
distribution, yet I tried looking on Debian server distribution, could
not find the package lc.

I have tried ./compile.sh and make, yet, too many troubles.

;;;; Compiling with clisp Common Lisp
clisp: /package/prog/clisp-clisp/lib/clisp-2.49+/full/lisp.run: No
such file or directory -- I don't know why.

Post by Pascal Bourguignon
https://gitlab.com/com-informatimago/com-informatimago
<https://gitlab.com/com-informatimago/com-informatimago> ; clone it
in ~/quicklisp/local-projects and use (ql:quickload
:com.informatimago.common-lisp.cesarum) to load it.

I have it already. I cannot load it, don't know where to start.

I do need cesarum, if you said it worked.

Post by Pascal Bourguignon
And to help, I write a Makefile that will invoke the various CL
implementations with the precise options to load
generate-executable.lisp correctly (it changes from one
implementation to another).
clisp -q -ansi -norc -x '(load "generate-executable.lisp")' -x '(quit)'
;; Loading file generate-executable.lisp ...
;; Loading file /Users/pjb/quicklisp/setup.lisp ...
;; Loading file /Users/pjb/quicklisp/ASDF.lisp ...
;; Loaded file /Users/pjb/quicklisp/ASDF.lisp
;; Loaded file /Users/pjb/quicklisp/setup.lisp
jean-cgi
; Loading "jean-cgi"
Bytes permanently allocated: 172,224
Bytes currently in use: 6,034,936
Bytes available until next GC: 1,507,604
;; Loaded file generate-executable.lisp
T
~/Sites/cgi

OK I see that is similar to my idea. I run the program in background,
which watches that file is saved. Yet, with or without Alexandria,
simple (exit) is not exiting in CGI, like Perl does.

Post by Pascal Bourguignon
So you can see how clisp is invoked to load generate-executable.lisp.
generate-executable.lisp loads quicklisp, and then use it to compile
and load jean-cgi. Finally it saves the executable image.
In more complex programs, generate-executable.lisp would first
delete the compiled fas files found in ~/.cache/common-lisp/*/… to
ensure that all the fas file loaded by quicklisp are freshly
compiled (notably with the *features* set that can be configured in
generate-executable.lisp).

OK thank you, I see, that is what I will implement.

Post by Pascal Bourguignon
Status: 302 Found
Location: http://localhost

That is what I need, yet now I need to know how to implement it.

Jean

[Pascal Bourguignon]

Post by Jean Louis

Post by Pascal Bourguignon
There is some confusion between the type of delivery you want, and
in the case of CGI, some technical difficulty in writing a file able
to do both.

I am just trying to understand, if there is no *standard-input*
coming, why the program is not going one step forward, to for example,
spit the headers our.

Post by Pascal Bourguignon
When you write a file starting with “#!” it’s to be considered as a
script by the unix system, that is, a file that will be processed
(interpreted) by the command given after “#!”.

Thank you for explanation, I am aware of it. Yet I am trying to run it
as a memory image, for the speed it is giving.

Post by Pascal Bourguignon
If you save the above file in test.cgi and chmod +x it, it won’t
work, because alexandria won’t be loaded by the script.

Right now, it works, as it is saved as memory image which itself has
Alexandria inside. I am using this small program to detect the change
when test.lisp is saved, so that it generates test.cgi which I can
then try to wget or browse.

In emacs, you can just type M-x compile RET when you save your source. (Assuming you wrote the Makefile as I advised).

Post by Jean Louis
(defparameter file (car *args*))

The expression in the defparameter form is evaluated when the source file (or the fas file) is loaded. It is NOT evaluated when the lisp image is loaded!


Also, you’ve been told to use *stars* around the name of special variables defined with defvar or defparameter. You really need to do it!

Post by Jean Louis
(defun check-the (file)
(let ((file (format nil "~a.lisp" file)))
(file-stat-mtime (file-stat file))))
(defparameter check (check-the file))
(defun do-compile (file) ;; (setf file "order")
(cd "/home/data1/protected/public_html/")
(let* ((original (format nil "~a.lisp" file))
(executable (format nil "~a.cgi" file))
(command (format nil "lisp -i ~a -x \"(saveinitmem \\\"~a\\\" :executable t :init-function 'main :quiet t :norc t)\"" original executable)))
(unless (equalp check (check-the file))
(print command)
(shell command)
(set-file-stat executable :mode #O0755)
(setf check (check-the file))))
(sleep .3))

Here, you’re not using clisp, but CMU CL (lisp).
(Each character is important, with computers).

Post by Jean Louis
(loop (do-compile file))
I can use it without Alexandria, I guess this is equivalent
below. Yet, I just need to find out how to recognize that there is no
*standard-input* coming, as it is more or less "hanging" if there is
no POST or GET parameter.

When you reach end-of-file. In general, the end of file “condition” can be dealt with in two different ways. Usually, I/O functions have optional parameters to let you choose how it will deal when it detects the end of a file.

Usually, the default way is to signal an END-OF-FILE error. The alternate way is to return a “guard” value (by default NIL), instead of the read object.

Post by Jean Louis
(setf custom:*ansi* t)
(defun POST-collect ()
(let (lines (loop for line = (read-line *standard-input* nil nil)
while line
collect))
(format "~{~a~}" lines)))

CL-USER> (defun POST-collect ()
(let (lines (loop for line = (read-line *standard-input* nil nil)
while line
collect))
(format "~{~a~}" lines)))


POST-COLLECT
CL-USER> (with-input-from-string (*standard-intput* "")
(post-collect))

*** - LET: illegal variable specification (LOOP FOR LINE = (READ-LINE *STANDARD-INPUT* NIL NIL) WHILE LINE COLLECT)


Perhaps you should read a tutorial about lisp programming before trying to implement your CGI script. I would advise:

http://clisp.hg.sourceforge.net/hgweb/clisp/clisp/raw-file/tip/doc/LISP-tutorial.txt

http://www.franz.com/resources/educational_resources/cooper.book.pdf (but for Allegro CL, you should ignore the parts that are implementation specific).

http://www.cliki.net/Online%20Tutorial

Post by Jean Louis
(defun headers nil

And you still use NIL instead of () for parameters.

Post by Jean Louis
(format t "Content-type: text/plain~2%"))

(defun main ()
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(POST-collect))))
;; (format t "Status: 302 Found~%Location: ~a~%~%" url)
(headers)
(princ "Hello there")
;; (finish-output)

I would advise you to keep calls to finish-output (and always use it when doing interactive I/O).

Post by Jean Louis
(exit 0)))
;; (main)

Post by Pascal Bourguignon
You could use a macro such as without-output or

Well thank you for that, I will keep it for later usage.
I was thinking that :verbose nil and :silent t within (ql:quickload ""
would suppress the messages. And in some scripts it does it so.
(ql:quickload "alexandria" :verbose nil :silent t)
without the output.

It seems to work. Notice however that it doesn’t catch *terminal-io* (it couldn’t do so conformingly, but neither without-output could).

Post by Jean Louis

Post by Pascal Bourguignon
By the way, you should normally use the nickname ALEXANDRIA, instead
of the specific versioned package name ALEXANDRIA.0.DEV, unless you
are using an internal API that you can expect to be specific to the
version 0 (development) of ALEXANDRIA.

That I did not know, for one reason, when I type alexandria on CLISP
REPL, it completes to alexandria.0.dev: and I was thinking that is
so. Now I see that I can use alexandria: only, thank you.

Post by Pascal Bourguignon
- a normal lisp source file,
- a lisp source file defining a package,
- an ASDF system definition file, to load your program with asdf or quicklisp,
- a lisp “script” to load and generate the executable image, and
- a Makefile to build it.
See attached files.
CL-USER> (ls)
./Makefile
./generate-executable.lisp
./jean-cgi.asd
./jean-cgi.lisp
./packages.lisp
; No value
CL-USER> (ql:quickload "jean-cgi")
jean-cgi
; Loading "jean-cgi"
("jean-cgi")
CL-USER>
The advantage of being able to load your program in an interactive
lisp image, is that you will be able to run and test it at the REPL,
with the help of the interactive debugger, etc. For example, this
CL-USER> (with-input-from-string (*standard-intput* "")
(jean-cgi:main))

Thank you, I am keeping that for later. Right now I have put
everything in one Lisp image, one order.lisp file, including the GnuPG
key, so that GnuPg directory is recreated on server automatically, and
that all inquiries are encrypted on the way back to me.

Your code is wrong. My code is correct: I proved it to you by showing you how it worked correctly in two different ways.

Now you can keep it for later, but then I will resume helping you only later.

Post by Jean Louis
Everything is working well so far, only that I cannot solve the
problem when the CGI is called directly, it is waiting too long.
#!/usr/bin/perl
exit;
there is no waiting time. And I came from Perl world.
(defun main ()
(exit))
and save as memory image, the wget is not exiting. Basically CLISP is
not exiting.

You’re lying.

Post by Jean Louis

Post by Pascal Bourguignon
CL-USER> (with-input-from-string (*standard-intput* "")
(contents-from-stream *standard-intput* :min-size 16384))
""
you might prefer using it rather than alexandria (but beware, I
license my software under the AGPL3 license).

That is best license currently for free software. I would use AGPL
myself, there is no incompatibility.
OK I have all your repository on my computer.
There is slight problem, there, if I come to directory, common-lisp,
and type make, thee command "lc" is missing. I don't have any
distribution, yet I tried looking on Debian server distribution, could
not find the package lc.
I have tried ./compile.sh and make, yet, too many troubles.

Don’t use them, they’re outdated. The point of cloning it in ~/quicklisp/local-projects is to be able to load it with quicklisp. I gave you the exact expression to use!

Post by Jean Louis
;;;; Compiling with clisp Common Lisp
clisp: /package/prog/clisp-clisp/lib/clisp-2.49+/full/lisp.run: No
such file or directory -- I don't know why.

Post by Pascal Bourguignon
https://gitlab.com/com-informatimago/com-informatimago
<https://gitlab.com/com-informatimago/com-informatimago> ; clone it
in ~/quicklisp/local-projects and use (ql:quickload
:com.informatimago.common-lisp.cesarum) to load it.

I have it already. I cannot load it, don't know where to start.

If you didn’t install it in ~/quicklisp/local-projects, then it will be harder.

Post by Jean Louis
I do need cesarum, if you said it worked.

Yes. That said, I left alexandria in your code that I modified to make it work. The limitation of alexandria shows only on string-stream, not on file-stream or (I assume) socket-stream.

Post by Jean Louis

Post by Pascal Bourguignon
And to help, I write a Makefile that will invoke the various CL
implementations with the precise options to load
generate-executable.lisp correctly (it changes from one
implementation to another).
clisp -q -ansi -norc -x '(load "generate-executable.lisp")' -x '(quit)'
;; Loading file generate-executable.lisp ...
;; Loading file /Users/pjb/quicklisp/setup.lisp ...
;; Loading file /Users/pjb/quicklisp/ASDF.lisp ...
;; Loaded file /Users/pjb/quicklisp/ASDF.lisp
;; Loaded file /Users/pjb/quicklisp/setup.lisp
jean-cgi
; Loading "jean-cgi"
Bytes permanently allocated: 172,224
Bytes currently in use: 6,034,936
Bytes available until next GC: 1,507,604
;; Loaded file generate-executable.lisp
T
~/Sites/cgi

OK I see that is similar to my idea. I run the program in background,
which watches that file is saved. Yet, with or without Alexandria,
simple (exit) is not exiting in CGI, like Perl does.

Post by Pascal Bourguignon
So you can see how clisp is invoked to load generate-executable.lisp.
generate-executable.lisp loads quicklisp, and then use it to compile
and load jean-cgi. Finally it saves the executable image.
In more complex programs, generate-executable.lisp would first
delete the compiled fas files found in ~/.cache/common-lisp/*/… to
ensure that all the fas file loaded by quicklisp are freshly
compiled (notably with the *features* set that can be configured in
generate-executable.lisp).

OK thank you, I see, that is what I will implement.

Post by Pascal Bourguignon
Status: 302 Found
Location: http://localhost

That is what I need, yet now I need to know how to implement it.

Look into the file I attached, instead of keeping them for later!!!

[Jean Louis]

Thank you.

Post by Pascal Bourguignon
Also, you’ve been told to use *stars* around the name of special
variables defined with defvar or defparameter. You really need to
do it!

Those are snippets from before, now is changing.

Post by Pascal Bourguignon

Post by Jean Louis
(defun check-the (file)
(let ((file (format nil "~a.lisp" file)))
(file-stat-mtime (file-stat file))))
(defparameter check (check-the file))
(defun do-compile (file) ;; (setf file "order")
(cd "/home/data1/protected/public_html/")
(let* ((original (format nil "~a.lisp" file))
(executable (format nil "~a.cgi" file))
(command (format nil "lisp -i ~a -x "(saveinitmem \"~a\" :executable t :init-function 'main :quiet t :norc t)"" original executable)))
(unless (equalp check (check-the file))
(print command)
(shell command)
(set-file-stat executable :mode #O0755)
(setf check (check-the file))))
(sleep .3))

Here, you’re not using clisp, but CMU CL (lisp).
(Each character is important, with computers).

I don't know what you wish to say with that one. I am using CLISP at
all times. CMU CL does not compile on 64 bit architectures

Thank you, I read all tutorials.

Post by Pascal Bourguignon

Post by Jean Louis
(format t "Content-type: text/plain~2%"))

(defun main ()
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(POST-collect))))
;; (format t "Status: 302 Found~%Location: ~a~%~%" url)
(headers)
(princ "Hello there")
;; (finish-output)

I would advise you to keep calls to finish-output (and always use it
when doing interactive I/O).

OK

Post by Pascal Bourguignon
Your code is wrong. My code is correct: I proved it to you by
showing you how it worked correctly in two different ways.

I have tried those files you have sent to me. I did not notice they
were attached until I did. So I tried. I get again the problem, of
little different nature. You can see on this screenshot:
Loading Image...

*** - OS-ERROR(EFAULT): Bad address
Break 1 [1]>

Post by Pascal Bourguignon

Post by Jean Louis
(defun main ()
(exit))
and save as memory image, the wget is not exiting. Basically CLISP is
not exiting.

You’re lying.

Maybe you are joking. Anyway, I am not lying. I am trying to find out
why is it, so that I can avoid it.

When you run that same jean-cgi script, as executable image and CGI on
your side, you don't get any error like me?


Jean

[Pascal Bourguignon]

Post by Jean Louis

Post by Pascal Bourguignon

Post by Jean Louis
(command (format nil "lisp -i ~a -x "(saveinitmem \"~a\" :executable t :init-function 'main :quiet t :norc t)"" original executable)))

Here, you’re not using clisp, but CMU CL (lisp).
(Each character is important, with computers).

I don't know what you wish to say with that one. I am using CLISP at
all times. CMU CL does not compile on 64 bit architectures

You have a typo, “lisp -i …” instead of “clisp -i …”.

Post by Jean Louis

Post by Pascal Bourguignon
Your code is wrong. My code is correct: I proved it to you by
showing you how it worked correctly in two different ways.

I have tried those files you have sent to me. I did not notice they
were attached until I did. So I tried. I get again the problem, of
https://rcdrun.com/files/tmp/2017-03-29-01:16:51.jpg
*** - OS-ERROR(EFAULT): Bad address
Break 1 [1]>

You must have new code.

Post by Jean Louis

Post by Pascal Bourguignon

Post by Jean Louis
(defun main ()
(exit))
and save as memory image, the wget is not exiting. Basically CLISP is
not exiting.

You’re lying.

Maybe you are joking. Anyway, I am not lying. I am trying to find out
why is it, so that I can avoid it.

(exit) will exit. In (defun main () (exit)), there’s no code doing any I/O.

Post by Jean Louis
When you run that same jean-cgi script, as executable image and CGI on
your side, you don't get any error like me?

Indeed, it works for me.

For example:


[***@despina :0.0]$ make
clisp -q -ansi -norc -x '(load "generate-executable.lisp")' -x '(quit)'
;; Loading file generate-executable.lisp ...
;; Loading file /Users/pjb/quicklisp/setup.lisp ...
;; Loading file /Users/pjb/quicklisp/ASDF.lisp ...
;; Loaded file /Users/pjb/quicklisp/ASDF.lisp
;; Loaded file /Users/pjb/quicklisp/setup.lisp
To load "jean-cgi":
Load 1 ASDF system:
jean-cgi
; Loading "jean-cgi"


Bytes permanently allocated: 172,224
Bytes currently in use: 6,043,064
Bytes available until next GC: 1,509,636
;; Loaded file generate-executable.lisp
T

~/Sites/cgi/jean
[***@despina :0.0]$ ./jean.cgi

~/Sites/cgi/jean
[***@despina :0.0]$ curl --head 'http://localhost/~pjb/cgi/jean.cgi'
HTTP/1.1 404 Not Found
Date: Tue, 28 Mar 2017 22:29:47 GMT
Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 OpenSSL/1.0.2k DAV/2
Content-Type: text/html; charset=iso-8859-1


~/Sites/cgi/jean
[***@despina :0.0]$ cat jean-cgi.lisp
(in-package "JEAN-CGI")

(defun main (&optional (arguments *args*))
(declare (ignore arguments))
(exit 0)
(let ((url "http://localhost")
(query (or (getenv "QUERY_STRING")
(alexandria:read-stream-content-into-string *standard-input*))))
(format t "Status: 302 Found~%Location: ~a~%~%" url)
(finish-output)
(exit 0)))



~/Sites/cgi/jean
[***@despina :0.0]$

The only difference as you can see, is that instead of getting a 302 status, we get a 404. This status is returned by apache that detects that the CGI didn’t do anything.

[Jean Louis]

Post by Pascal Bourguignon

Post by Jean Louis

Post by Pascal Bourguignon

(command (format nil "lisp -i ~a -x "(saveinitmem "~a" :executable t :init-function 'main :quiet t :norc t)"" original executable)))

Here, you’re not using clisp, but CMU CL (lisp).
(Each character is important, with computers).

I don't know what you wish to say with that one. I am using CLISP at
all times. CMU CL does not compile on 64 bit architectures

You have a typo, “lisp -i …” instead of “clisp -i …”.

Because I have either lisp image with quicklisp inside, named lisp, or
symlink sometimes.

Post by Pascal Bourguignon

Post by Jean Louis
*** - OS-ERROR(EFAULT): Bad address
Break 1 [1]>

You must have new code.

Nothing changed in your code. I just changed the directory where
quicklisp is located, and did the make, it makes the jean.cgi, I copy
it to test.cgi in ~/public_html and try to call it, it hangs for some
time, until I see this screenshot, but now I did not get error code,
just hanging longer time:
Loading Image...

I tried with few webservers. When you say that Apache is delivering
404, and recognizing no output, I will try with other servers
too. Maybe that is the problem.

Jean

[Pascal Bourguignon]

Post by Jean Louis

Post by Pascal Bourguignon

Post by Jean Louis

Post by Pascal Bourguignon

(command (format nil "lisp -i ~a -x "(saveinitmem "~a" :executable t :init-function 'main :quiet t :norc t)"" original executable)))

Here, you’re not using clisp, but CMU CL (lisp).
(Each character is important, with computers).

I don't know what you wish to say with that one. I am using CLISP at
all times. CMU CL does not compile on 64 bit architectures

You have a typo, “lisp -i …” instead of “clisp -i …”.

Because I have either lisp image with quicklisp inside, named lisp, or
symlink sometimes.

ok.

Post by Jean Louis

Post by Pascal Bourguignon

Post by Jean Louis
*** - OS-ERROR(EFAULT): Bad address
Break 1 [1]>

You must have new code.

Nothing changed in your code. I just changed the directory where
quicklisp is located, and did the make, it makes the jean.cgi, I copy
it to test.cgi in ~/public_html and try to call it, it hangs for some
time, until I see this screenshot, but now I did not get error code,
https://rcdrun.com/files/tmp/2017-03-29-01:37:35.jpg <https://rcdrun.com/files/tmp/2017-03-29-01:37:35.jpg>

Then you have to debug it.
Have a look at http://www.cliki.net/TutorialClispDebugger <http://www.cliki.net/TutorialClispDebugger>
and try to find what function returned this error.

Post by Jean Louis
I tried with few webservers. When you say that Apache is delivering
404, and recognizing no output, I will try with other servers
too. Maybe that is the problem.

[Jean Louis]

Post by Pascal Bourguignon
Then you have to debug it.
Have a look at http://www.cliki.net/TutorialClispDebugger <http://www.cliki.net/TutorialClispDebugger>
and try to find what function returned this error.

I did not find any reference on how to debug over HTTP. I would like
to find the cause or find out how to avoid it.

In console it is giving redirect. On HTTP, it is hanging for longer
time, then I get like now, in this attempt:

admin-> wget -O /dev/shm/test "http://localhost/test.cgi"; less /dev/shm/test
--2017-03-29 01:56:37-- http://localhost/test.cgi
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified
Saving to: ‘/dev/shm/test’

/dev/shm/test [ <=> ] 38 --.-KB/s in 0s

2017-03-29 01:57:13 (3.93 MB/s) - ‘/dev/shm/test’ saved [38]

*** - Ctrl-C: User break
Break 1 [1]>

[Pascal Bourguignon]

Post by Jean Louis

Post by Pascal Bourguignon
Then you have to debug it.
Have a look at http://www.cliki.net/TutorialClispDebugger <http://www.cliki.net/TutorialClispDebugger>
and try to find what function returned this error.

I did not find any reference on how to debug over HTTP. I would like
to find the cause or find out how to avoid it.
In console it is giving redirect. On HTTP, it is hanging for longer
admin-> wget -O /dev/shm/test "http://localhost/test.cgi"; less /dev/shm/test
--2017-03-29 01:56:37-- http://localhost/test.cgi
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified
Saving to: ‘/dev/shm/test’
/dev/shm/test [ <=> ] 38 --.-KB/s in 0s
2017-03-29 01:57:13 (3.93 MB/s) - ‘/dev/shm/test’ saved [38]
*** - Ctrl-C: User break
Break 1 [1]>

Well, it looks like test.cgi was hung, not printing anything, and that your web server decided to kill it with a SIGINT. This was interpreted as a C-c by clisp, and entered into the debugger, then the web server took that output and returned it.

Here’s how I catch and report errors, saving them into files:



I’ve left the call to (error "Simulated error.”) for your tests.
You must remove it (and recompile with make) before trying it for real.

In this case, we want to catch the errors before issuing the headers, so I collect them into a string errout, and then depending on whether there was an error to compute query, I print an error web page, or perform the normal processing.


[***@despina :0.0 jean]$ rm /tmp/jean-cgi.*
[***@despina :0.0 jean]$ make
make: Nothing to be done for `all'.
[***@despina :0.0 jean]$ ./jean.cgi </dev/null
Content-Type: text/plain


2017-03-29 02:04:20

<1/89> #<system-function show-stack> 3
<2/82> #<compiled-function system::print-backtrace>
<3/78> #<compiled-function jean-cgi::print-backtrace>
[64] frame binding variables (~ = dynamically):
| ~ *print-level* <--> nil
Printed 3 frames
ERROR while ((setf jean-cgi::query (or (error "Simulated error.") (getenv "QUERY_STRING") (alexandria.0.dev:read-stream-content-into-string *standard-input*)))):
Simulated error.

[***@despina :0.0 jean]$ cat /tmp/jean-cgi.trace
Found error Simulated error.

2017-03-29 02:04:20

<1/89> #<system-function show-stack> 3
<2/82> #<compiled-function system::print-backtrace>
<3/78> #<compiled-function jean-cgi::print-backtrace>
[64] frame binding variables (~ = dynamically):
| ~ *print-level* <--> nil
Printed 3 frames
ERROR while ((setf jean-cgi::query (or (error "Simulated error.") (getenv "QUERY_STRING") (alexandria.0.dev:read-stream-content-into-string *standard-input*)))):
Simulated error.

[***@despina :0.0 jean]$ cat /tmp/jean-cgi.errors

2017-03-29 02:04:20

<1/89> #<system-function show-stack> 3
<2/82> #<compiled-function system::print-backtrace>
<3/78> #<compiled-function jean-cgi::print-backtrace>
[64] frame binding variables (~ = dynamically):
| ~ *print-level* <--> nil
Printed 3 frames
ERROR while ((setf jean-cgi::query (or (error "Simulated error.") (getenv "QUERY_STRING") (alexandria.0.dev:read-stream-content-into-string *standard-input*)))):
Simulated error.

[***@despina :0.0 jean]$ wget --save-headers http://localhost/~pjb/cgi/jean/jean.cgi -O /tmp/out
--2017-03-29 02:04:50-- http://localhost/~pjb/cgi/jean/jean.cgi
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: '/tmp/out'

/tmp/out [ <=> ] 68 --.-KB/s in 0.005s

2017-03-29 02:04:50 (14.5 KB/s) - '/tmp/out' saved [68]

[***@despina :0.0 jean]$ cat /tmp/out
HTTP/1.1 200 OK
Date: Wed, 29 Mar 2017 00:04:50 GMT
Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 OpenSSL/1.0.2k DAV/2
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain

*** - UNIX error 13 (EACCES): Permission denied
Exiting on signal 6

This error is due to the fact that the web server doesn’t run the CGI under the same user as yourself, so the files created don’t have the right access rights and owner. Let’s remove them:

[***@despina :0.0 jean]$ sudo rm /tmp/jean-cgi.*

and try again:

[***@despina :0.0 jean]$ wget --save-headers http://localhost/~pjb/cgi/jean/jean.cgi -O /tmp/out
--2017-03-29 02:05:04-- http://localhost/~pjb/cgi/jean/jean.cgi
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: '/tmp/out'

/tmp/out [ <=> ] 441 --.-KB/s in 0s

2017-03-29 02:05:04 (32.4 MB/s) - '/tmp/out' saved [441]

[***@despina :0.0 jean]$ cat /tmp/out
HTTP/1.1 200 OK
Date: Wed, 29 Mar 2017 00:05:04 GMT
Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 OpenSSL/1.0.2k DAV/2
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/plain


2017-03-29 02:05:04

<1/89> #<system-function show-stack> 3
<2/82> #<compiled-function system::print-backtrace>
<3/78> #<compiled-function jean-cgi::print-backtrace>
[64] frame binding variables (~ = dynamically):
| ~ *print-level* <--> nil
Printed 3 frames
ERROR while ((setf jean-cgi::query (or (error "Simulated error.") (getenv "QUERY_STRING") (alexandria.0.dev:read-stream-content-into-string *standard-input*)))):
Simulated error.

[***@despina :0.0 jean]$

Works nicely, the error is detected and reported.
After having removed the call to error, it works as expected:

[***@despina :0.0 jean]$ sudo rm /tmp/jean-cgi.*
Password:
[***@despina :0.0 jean]$ ./jean.cgi </dev/null
Status: 302 Found
Location: http://localhost


Got query: ""
at url: "http://localhost”

Again, remove the files created when you switch between shell runs and cgi runs:

[***@despina :0.0 jean]$ sudo rm /tmp/jean-cgi.*

[***@despina :0.0 jean]$ wget --save-headers http://localhost/~pjb/cgi/jean/jean.cgi -O /tmp/out
--2017-03-29 02:09:49-- http://localhost/~pjb/cgi/jean/jean.cgi
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://localhost [following]
--2017-03-29 02:09:49-- http://localhost/
Reusing existing connection to [localhost]:80.
HTTP request sent, awaiting response... 200 OK
Length: 44 [text/html]
Saving to: '/tmp/out'

/tmp/out 100%[===================================================>] 44 --.-KB/s in 0s

2017-03-29 02:09:49 (8.39 MB/s) - '/tmp/out' saved [44/44]

[***@despina :0.0 jean]$ cat /tmp/out
HTTP/1.1 200 OK
Date: Wed, 29 Mar 2017 00:09:49 GMT
Server: Apache/2.2.32 (Unix) mod_ssl/2.2.32 OpenSSL/1.0.2k DAV/2
Last-Modified: Tue, 28 Mar 2017 18:22:36 GMT
ETag: "168d58e-2c-54bce8b4d5300"
Accept-Ranges: bytes
Content-Length: 44
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: text/html

<html><body><h1>Welcome!</h1></body></html>
[***@despina :0.0 jean]$

[Jean Louis]

Post by Pascal Bourguignon

Post by Jean Louis
*** - Ctrl-C: User break
Break 1 [1]>

Well, it looks like test.cgi was hung, not printing anything, and
that your web server decided to kill it with a SIGINT. This was
interpreted as a C-c by clisp, and entered into the debugger, then
the web server took that output and returned it.

Yes, that is what I assume.

Post by Pascal Bourguignon
I’ve left the call to (error "Simulated error.”) for your tests.
You must remove it (and recompile with make) before trying it for real.
In this case, we want to catch the errors before issuing the
headers, so I collect them into a string errout, and then depending
on whether there was an error to compute query, I print an error web
page, or perform the normal processing.

Thank you much.
Good to know, I will be using it now.

Post by Pascal Bourguignon
<1/89> #<system-function show-stack> 3
<2/82> #<compiled-function system::print-backtrace>
<3/78> #<compiled-function jean-cgi::print-backtrace>
| ~ *print-level* <--> nil
Printed 3 frames
Simulated error.
Works nicely, the error is detected and reported.
Status: 302 Found
Location: http://localhost
Got query: ""
at url: "http://localhost”

So it works for you?

I have tried now with 3-4 web servers.

If I let this like following:

(let ((errout (with-output-to-string (*error-output*)
(reporting-errors
(setf query (or (getenv "QUERY_STRING")
(alexandria:read-stream-content-into-string *standard-input*)))))))

I do not wait this time long, maybe 3-4 seconds, then I get:

wget -O /dev/shm/test "http://localhost/test.cgi"; less /dev/shm/test
--2017-03-29 10:46:46-- http://localhost/test.cgi
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2017-03-29 10:46:53 ERROR 500: Internal Server Error.

and /tmp/jean-cgi.trace is empty, no other file.

What is wrong on my side that is different to your side? It works on
your side, and not on my side.

Jean

[Jean Louis]

Thank you for files, that is good for my education.

Jean

[Jean Louis]

Thank you for helping.

I found solution, as jasom on IRC, told me to check for
*standard-input* by using listen.

So, now I can run the CGI without POST and without GET parameters, and
without getting blocked, as Lisp is not waiting any more.

In the mean time I am learning ASDF.

Jean

(load "/home/data1/protected/lisp/quicklisp.lisp")
(ql:quickload "alexandria")

(defun main (&optional (arguments *args*))
(if (listen *standard-input*)
(progn
(headers)
(print "Standard input here"))
(progn
(headers)
(print "No standard input")))

;; (declare (ignore arguments))
;; (unless (listen *standard-input*)
;; ;; (getenv "QUERY_STRING"))
;; (let ((url "http://localhost")
;; (query (or (getenv "QUERY_STRING")
;; (alexandria:read-stream-content-into-string *standard-input*))))))
;; (format t "Status: 302 Found~%Location: ~a~%~%" url)

(finish-output)
(exit 0))

[Jean Louis]

Tip.
Lispers (well, at least one of them) abhor “main” :) :) :)

Mi spiega quello.

Explain me.

[Antoniotti Marco]

Ti spiego :)

“main” is a C-ism (Java-ism), which is also indicating that the program must be run a “script”. Lispers (at least one of them) like rolling in mud :)

Cheers
—
MA

Post by Jean Louis

Tip.
Lispers (well, at least one of them) abhor “main” :) :) :)

Mi spiega quello.
Explain me.

--
Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01
DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it
Viale Sarca 336
I-20126 Milan (MI) ITALY

Please check: http://cdac2017.lakecomoschool.org
Please check: https://sites.google.com/site/troncopackage/

Please note that I am not checking my Spam-box anymore.
Please do not forward this email without asking me first (cum grano salis).

[Jean Louis]

Post by Antoniotti Marco
Ti spiego :)
“main” is a C-ism (Java-ism), which is also indicating that the
program must be run a “script”. Lispers (at least one of them) like
rolling in mud :)

Not that I understand. I am using "main" in some programs, in others
not, it depends. I can just guess it is not hard coded to be
executed. But I don't know.

[Pascal Bourguignon]

Post by Jean Louis
Thank you for helping.
I found solution, as jasom on IRC, told me to check for
*standard-input* by using listen.
So, now I can run the CGI without POST and without GET parameters, and
without getting blocked, as Lisp is not waiting any more.
In the mean time I am learning ASDF.
Jean
(load "/home/data1/protected/lisp/quicklisp.lisp")
(ql:quickload "alexandria")
(defun main (&optional (arguments *args*))
(if (listen *standard-input*)
(progn
(headers)
(print "Standard input here"))
(progn
(headers)
(print "No standard input")))
;; (declare (ignore arguments))
;; (unless (listen *standard-input*)
;; ;; (getenv "QUERY_STRING"))
;; (let ((url "http://localhost")
;; (query (or (getenv "QUERY_STRING")
;; (alexandria:read-stream-content-into-string *standard-input*))))))
;; (format t "Status: 302 Found~%Location: ~a~%~%" url)
(finish-output)
(exit 0))

Neither using LISTEN or directly READ-SEQUENCE are good solutions.
Either one will work only by chance.

LISTEN in particular will fail, if for whatever reason, the web server didn’t start writing the PUT data or POST parameters to the CGI pipe, when the CGI calls LISTEN. I guess the same is true with READ-SEQUENCE: it won’t read an end-of-file if the web server doesn’t close that pipe.

And let me you ask the question: what reason does the web server have to close that pipe if it didn’t have to write anything to it, because it was a GET request?

The correct solution, as always, is to read the documentation, specifically, the RFC 3875 <https://tools.ietf.org/html/rfc3875>
Or you may browse the Wikipedia summary, and find out that the CGI specifies that the environment variable REQUEST_METHOD will indicate whether it’s a GET, a PUT, a POST or whatever else. You must test this before trying to read stdin!


(defun main (&optional (arguments *args*))
(let ((method (getenv "REQUEST_METHOD")))
(cond ((or (string= method "PUT")
(string= method "POST"))
(headers)
(print "Standard input here"))
((string= method "GET")
(headers)
(print "No standard input"))
(t
(headers)
(print "Method not handled")))))

[Jean Louis]

Post by Pascal Bourguignon
Neither using LISTEN or directly READ-SEQUENCE are good solutions.
Either one will work only by chance.

I guess that is deeper foundation of how it works. On my side it works
now well, so it will work in practice.

Post by Pascal Bourguignon
LISTEN in particular will fail, if for whatever reason, the web
server didn’t start writing the PUT data or POST parameters to the
CGI pipe, when the CGI calls LISTEN. I guess the same is true with
READ-SEQUENCE: it won’t read an end-of-file if the web server
doesn’t close that pipe.

OK, maybe, I don't know, I cannot say. I just test it on 2-3
webservers, those I use also remotely, if it works locally, it works
remotely, then I leave it working for years.

Post by Pascal Bourguignon
And let me you ask the question: what reason does the web server
have to close that pipe if it didn’t have to write anything to it,
because it was a GET request?
The correct solution, as always, is to read the documentation,
specifically, the RFC 3875 <https://tools.ietf.org/html/rfc3875>
Or you may browse the Wikipedia summary, and find out that the CGI
specifies that the environment variable REQUEST_METHOD will indicate
whether it’s a GET, a PUT, a POST or whatever else. You must test
this before trying to read stdin!
(defun main (&optional (arguments *args*))
(let ((method (getenv "REQUEST_METHOD")))
(cond ((or (string= method "PUT")
(string= method "POST"))

Thank you.

I have just shown the sample, in reality I am testing for QUERY_STRING
to get the GET parameters, if none, I test for *standard-input*, and
later for certain parameters, if they are missing, it goes to error
handling.

Jean

[Pascal Bourguignon]

Post by Jean Louis

Post by Pascal Bourguignon
Neither using LISTEN or directly READ-SEQUENCE are good solutions.
Either one will work only by chance.

I guess that is deeper foundation of how it works. On my side it works
now well, so it will work in practice.

It will work until the system is loaded and the web server is interrupted between fork/execing the CGI, and writing the parameter.

Post by Jean Louis

Post by Pascal Bourguignon
LISTEN in particular will fail, if for whatever reason, the web
server didn’t start writing the PUT data or POST parameters to the
CGI pipe, when the CGI calls LISTEN. I guess the same is true with
READ-SEQUENCE: it won’t read an end-of-file if the web server
doesn’t close that pipe.

OK, maybe, I don't know, I cannot say. I just test it on 2-3
webservers, those I use also remotely, if it works locally, it works
remotely, then I leave it working for years.

Good luck! Be sure to put your name on any system you program like that, so I don’t use them and don’t risk a painful death.

Post by Jean Louis

Post by Pascal Bourguignon
And let me you ask the question: what reason does the web server
have to close that pipe if it didn’t have to write anything to it,
because it was a GET request?
The correct solution, as always, is to read the documentation,
specifically, the RFC 3875 <https://tools.ietf.org/html/rfc3875>
Or you may browse the Wikipedia summary, and find out that the CGI
specifies that the environment variable REQUEST_METHOD will indicate
whether it’s a GET, a PUT, a POST or whatever else. You must test
this before trying to read stdin!
(defun main (&optional (arguments *args*))
(let ((method (getenv "REQUEST_METHOD")))
(cond ((or (string= method "PUT")
(string= method "POST"))

Thank you.
I have just shown the sample, in reality I am testing for QUERY_STRING
to get the GET parameters, if none, I test for *standard-input*, and
later for certain parameters, if they are missing, it goes to error
handling.

QUERY_STRING is present when REQUEST_METHOD=POST !
And there is an intersection of values for QUERY_STRING for both GET and POST.

[Jean Louis]

Post by Pascal Bourguignon
QUERY_STRING is present when REQUEST_METHOD=POST !
And there is an intersection of values for QUERY_STRING for both GET and POST.

That is true. Never mind, I am not even using GET.

Thank you.