Syntax highlighting nightmare
Home About Perl Subversion Training Talks Articles Blog Music Download Contact
Blog entries
2009 Jan 01

Perl 6 Cookbook.
2009 Jan 01

New Year's Resolutions.
2008 Dec 30

PPI based Syntax highlighting for Perl 5.
2008 Dec 29

Syntax highlighting for Perl 6.
2008 Dec 11

Plans for Integrating Padre with Parrot and Rakudo.
2008 Dec 10

Grant accepted for Integrating Padre with Parrot and Rakudo.
2008 Dec 10

Plans for the next month or two.
2008 Dec 05

Perlshere.
2008 Nov 30

Portable Padre 0.19 for Windows.
2008 Nov 27

10-fold grows in Padre user base.
2008 Nov 26

How many test harnesses are too many?.
2008 Nov 25

Licenses on CPAN. Again.
2008 Nov 20

Padre talk in Haifa, reality check.
2008 Nov 17

Padre 0.17 was released.
2008 Nov 12

Talking about Padre and wxPerl in Haifa.
2008 Nov 11

Backlinks or links back to your site.
2008 Nov 10

Building your resume.
2008 Nov 09

How to run an Open Source Project.
2008 Nov 06

Syntax highlighting nightmare.
2008 Nov 04

2008Q4 TPF Grant Proposals.
2008 Nov 02

Subversion committer statistics.
2008 Oct 28

Perl Application Development and Distribution Platform.
2008 Oct 28

Compare Languages by usage.
2008 Oct 23

yak shaving.
2008 Oct 21

Recursive development that leads nowhere.
2008 Oct 18

Licenses in META.yml on CPAN.
2008 Oct 17

Shall I enable some form of trackback or commenting?.
2008 Oct 15

Shana Tova - New Year's resolution.
2008 Oct 15

Perl needs is_number and similar functions (nearly built in).
2008 Sep 22

The Quest for the Perfect Editor.
2008 Sep 04

Living on the border.
2008 Sep 02

TAP - Test Anything Protocol.
2008 Aug 31

Padre - the journey I..
2008 Aug 21

Who needs an IDE for Perl anyway?.
2008 Aug 09

Padre project web site.
2008 Jul 27

Padre.
2008 Jul 23

White Camel.
2008 Jul 18

Name a Perl IDE - get a Perl book or YAPC ticket.
2008 Jul 09

QA Hackathon in Israel.
2008 Jul 01

OSDC Israel 2009 - Call for organizers.
2008 Jun 11

Selenium on Ubuntu 8.04 (Hardy).
2008 Jun 09

Testing Hello World.
2008 Jun 08

Wifi is working again!.
2008 Jun 07

CPANTS update.
2008 Jun 04

Frequent Internet blackouts.
2008 Jun 03

Upgrading to Ubuntu 8.04 Hardy on Compaq (HP) nc6400. .
2008 May 24

Test Automation Tips.
2008 May 22

Open Source IDE for Perl.
2008 May 21

This week in Ruby.
2008 May 21

Being included on Planet Perl.
2008 May 14

Adding tag cloud to the blog.
2008 May 14

Ubuntu 7.04 (beta) Feisty Fawn on Compaq (HP) nc6400.
2008 May 13

Test automation using Perl master class in Chicago.
2008 May 13

Adding tags to the blog.
2008 May 09

Automated Testing in PHP, Python, Ruby and Perl.
2008 Apr 03

Strawberry Perl for Windows.
2008 Apr 01

Oslo Hackathon day -4.
2008 Mar 28

Blogging about Perl outside the community?.
2008 Mar 27

OSCON Proposals rejected.
2008 Mar 26

Preparing for the QA Hackathon in Oslo.
2008 Mar 25

Missing licenses on CPAN modules?.
2008 Mar 24

License of Perl Modules on CPAN.
2007 Dec 24

Joining Technorati?.
2007 Dec 24

Regular Expressions in Perl 5.10.
2007 Dec 24

Switching in Perl 5.10.
2007 Dec 24

Smart Matching in Perl 5.10.
2007 Dec 24

What's new in Perl 5.10? say, //, state.
2007 Dec 23

The Zulo interview was published.
2007 Dec 08

Frequency of programming languages on LinkedIn.
2007 Dec 06

Interview in Zulo.
2007 Dec 06

Sun Startup Essentials Launch.
2007 Aug 25

Testing PostgresSQL.
2007 Aug 25

Testing Pugs and Perl 6.
2007 Aug 22

Testing Ruby.
2007 Aug 22

Testing GHC, the Glasgow Haskell Compiler.
2007 Aug 22

Testing NUT, the Network UPS Tools.
2007 Aug 21

Testing SQLite .
2007 Aug 20

Smoked Parrot.
2007 Aug 20

Quality Assurance of Perl 5.
2007 Jul 09

Using mod_perl for szabgab.com.
2007 Jul 07

Quality Assurance and Automated Testing in Open Source Software.
2007 Jul 07

Add tags to CPAN modules via CPAN::Forum .
2007 Jun 15

Windows on VMware.
2007 Jun 13

Reducing the social gap of the information age.
2007 May 25

Moving to a new server.
2007 May 04

Preparing an application for distribution.
2007 May 01

Spreadsheet::ParseExcel is looking for a maintainer.
2007 Apr 28

CPAN Modules in Linux Distributions.
2007 Apr 18

Version control of single files using Subversion.
2007 Apr 13

Testing results, Perl and CPAN module availability.
2006 Aug 05

Perltraining.org split into two.
2006 Jul 23

Upgrading Ubuntu to 6.06, (Dapper Drake).
2006 Jul 22

Ginger Spam Salad.
2006 Jul 20

Automating the blog.
2006 Jul 19

Wish list: search engine for Perl related sites.
2006 Jul 19

Perltraining.org .
2006 Jul 19

More blog related issues.
2006 Jul 19

Starting a blog.

 
home | blog

Syntax highlighting nightmare

Published on 2008.11.06 at 06:41:14

Tags: perl, PPI, syntax highlighting, Padre

Two days ago I got, what is probably the most serious bug report so far for Padre. As it turns out while we are trying to build the best Perl IDE on Earth or whatever big title, we cannot even parse perl 5.10 correctly. Specifically it does not know what to do with the new defined-or // operator.

By chance just a few days earlier I wrote down the version numbers of our major non-perl dependencies, that is wxWidgets and Scintilla. This information now helped me to locate the problem and try to evaluate how can we solve it.

The latest version of Alien::wxWidgets on CPAN is 0.40 released on Oct 15, 2008. Based on this file I understand that it is using version 2.8.8 of wxWidgets. Checking the contrib/src/stc/scintilla/README.txt file in the wxWidgets 2.8.8 source code reveals that it uses Scintilla 1.70.

According to the wxWidgets web site currently the latest version of wxWidgets is 2.8.9 that was released on Sept 22, 2008. Based on file same README file we can find out that it also uses version 1.70 of Scintilla.

According to Scintilla web site the latest version of Scintilla is 1.77. It was released on Oct 18, 2008 while 1.70 was released 20 June 2006, more than 2 years ago.

Our problem is that perl 5.10 came out on Dec 18, 2007 and Scintilla included support for the new syntax only in version 1.76. (See the Scintilla History.

Solution 1

So the immediate idea was to check how can we get the latest Scintilla.

I checked the wxWidgtes development, and in their trunk, they indeed have a newer version of Scintilla. According to README in the trunk of wxWidgtes, it currently contains Scintilla 1.75. Argh.

That's still too old. In order for Padre to support perl 5.10, first Scintilla 1.77 needs to be integrated into wxWidgets. Then we have to wait for the next release of wxWidgetes and then wait some more till wxPerl catches on. By the time all this is done, a newer version of Perl will come out with yet another keyword or construct and we are again in a awful big delay.

Solution 2

We could also sidestep the wxWidgets integration of Scintilla and integrate it ourself but that means an awful lot of XS work that none of us likes and we are still dependent on Scintilla.

Solution 3 - write our own syntax highlighter

In the end no matter how good these syntax highlighters are, there are still going to be cases when they fail. After all only perl can parse Perl.

Well, except of course PPI.

Indeed, yesterday morning, just a few hours after we discussed the situation on #padre, Fayland Lam has already implemented a solution.

Of course it was slow as Adam Kennedy promised but it worked.

Some tweaking and now it is available as an experimental feature.

TODO

We still need to solve the speed issue though.

A quick analysis using Devel::NYTProf shows that the biggest time consumption is PPI analysing the file.

Two directions where people might want to help:

  • Implement more of PPI::XS to make PPI fast.
  • Change PPI so it can parse only part of the file and then restart later from that point.

Other ideas are also welcome.

Comments?

Instead of enabling comments here, please write your comment in your own blog and send me the link or paste it here:

Enter the link to the permalink of your response:

Trackbacks will show up only after manual approval.
Last Update: Tue Sep 25 17:06:26 2007
© 2004-2008 Gabor Szabo