Re: [O] Citation processing via Zotero + zotxt

[Richard Lawrence]

Hi John and all,

Thanks for your input!

John Kitchin <address@hidden> writes:

> This really means you are using bibtex as the reference database
> backend, and probably helm-bibtex to insert citations as org-mode links.
> Bibliography generation is probably managed by LaTeX/Bibtex. org-ref
> mostly provides the functionality of the links, and some utility
> functions to help build and maintain bibtex files, and to check all the
> cross-reference, label and citation links in the document.
>
> Ideally, something like zotxt would enable Zotero as the reference
> database backend with some selection mechanism (it could be helm, or
> ido, or some other interface) to insert citation syntax

This is pretty much what the zotxt-emacs package does.  It communicates
with the zotxt API, and it defines a minor mode for use with Org.  This
minor mode provides functions for querying the zotxt database (by
default, it searches by author, title and year) and for inserting and
updating Org links that point to items in the Zotero database.

It should be straightforward to extend these capabilities to insert
citation syntax instead of, or in addition to, links.

> , and at export time also leverage Zotero to generate the
> bibliography. I guess this would not too often be export to LaTeX,
> since that would require syncing with a Bibtex file. For quick and
> dirty work this would be fine, but my experience is converted bibtex
> files are never clean enough for production publication, and require a
> lot of manual cleaning of capitalization, journal abbreviations, math,
> etc... It is ok to do it once, but not every time you build a
> document!

Hmm, that is interesting and a bit concerning.  Zotero has the ability
to export sets of items to a BibTeX file.  Ideally this kind of cleanup
should, and in principle could, be dealt with `upstream' in the Zotero
database.  The CSL representation of items allows for pretty
fine-grained and explicit control over things like capitalization and
abbreviations.  But I do not know how much of that is exposed in Zotero,
and I don't know how well it is preserved upon export to BibTeX.

Certainly, I'd want export to LaTeX to be just as quick and painless as
export to other backends in general.  Some manual cleanup of the .bib
file seems acceptable to me as a final step before publication, but I
think it would count against Zotero if it was too annoying to

Zotero can also import BibTeX, so it would also be possible in principle
to go in the other direction: keep your database as .bib, but convert it
on the fly when exporting to other backends.

> A minimal, generalized interface that would probably work with org-ref
> would look something like (* is some prefix name like org-cite-):
>
> *-select-citations-from-db opens some selection interface with
>   candidates from the db-backend ((org-)bibtex, Zotero, Mendeley, RIS,
>   ...), allows user to select some citations with some filtering
>   capability then inserts citation syntax into org file. This function
>   should allow multiple selections, and allow appending to an existing
>   citation. org-ref supports reftex and helm-bibtex for this, and ebib
>   would also work.

As I mentioned above, zotxt-emacs already provides an API in this direction.

> *-export-citations This is the most difficult part, this code would have
>   to functionally do what LaTeX/BibTex accomplishes, and probably with
>   multiple passes to collect citations, get formatted bibliographic
>   entries, replace citations in the text (eg with numbers, au-year,
>   etc...), and insert the formatted bibliography in the right place. If
>   Zotero can give the formatted entry for each citation, this might be a
>   good start. It would be integrated into the export process. It might
>   even be backend-agnostic if the formatting was just plain or in org
>   markup text.

Yep, this is the hard one, but there is some more-or-less working code
to do this (via my org-citeproc script) in the in-progress org-cite.el.
I'm proposing reworking this code to communicate with Zotero, rather
than org-citeproc or another citeproc implementation.

> *-on-cite-click-func This is technically optional, but it is what makes
>   org-ref so great. Figures out what citation was clicked on, and runs
>   some action on that citation, e.g. open citation, find related
>   articles in google scholar, open browser to citation url, open notes
>   to the citation, etc... This is org-ref's most clever code, figuring
>   out what was clicked on, and providing multiple actions on that
>   citation key.

This needs more work, in org-element.  Right now, as far as I know,
there's no support for making citation-reference objects clickable.  On
the other hand, it doesn't seem like it should be that difficult.

> It is pretty easy to use zotero. I couldn't figure out zotxt though.

Hmm, where did you have trouble?  I was able to install it via this link

https://addons.mozilla.org/en-US/firefox/addon/zotxt/

and that was pretty much all there was to do.  (Figuring out how to use
the API it exposes is another matter, but that's programming work, not
something an ordinary user would have to do.)

>> Maybe two way sync between BibTeX and Zotero would be sufficient for
>> power users? I know there is already automatic exporters to BibTex for
>> Zotero, but some work needs to be done to make it a two-way sync.
>
> I am personally skeptical of this, having "translated" Endnote libraries
> to Bibtex and back. The simple stuff mostly works, but accented names,
> chemical formulas, and many other things tend to not translate very well.

Me too.  I think two-way sync is probably too annoying to try to get
right, and there isn't much to be gained by it.  I think instead we
should just support keeping reference information in a single
database---even if we "bless" several different kinds of database
(Zotero, org-bibtex, .bib)---and then provide a way to export citations
that point to items in that database for all relevant export backends.

Eric's question is relevant here:

Eric S Fraga <address@hidden> writes:

> 2. How would I use this starting from an org-bibtex database (which I
>    typically export to bibtex)?

I can envision a couple of possibilities.  One simple option would be to
switch to managing your reference database with Zotero, by exporting
from org-bibtex to .bib, and then importing the .bib into Zotero.

I don't want to force that on anyone, though.  Another option is to
use the org-bibtex to produce .bib at export time, and then use Zotero
to read the .bib and process citations when exporting to non-LaTeX formats.

This second option is more work, as I don't know of any API for loading
items into Zotero's citation processor in BibTeX format.  But given that
Zotero is able to import .bib files, I imagine this API would not be too
much work to build.

So, the short answer is: I think you should be able to continue to use
org-bibtex as your reference database, and gain citation support when
exporting to non-LaTeX backends, perhaps with as little effort as
installing Zotero + zotxt and writing

#+BIBDB: my-org-bibtex-file.org

in your document.  Anyway, that would be my goal.
 
Best,
Richard