Television is the most favorite pastime activity of the world. Remarkably, it has so far ignored the digital revolution; the way we watch television hasn’t changed since its invention 75 years ago. But the time of passive TV consumption may be over soon: Advances in video acquisition technology, novel image analysis algorithms, and the pace of progress in computer graphics hardware together drive the development of a new type of visual entertainment medium. The scientific and technological obstacles towards realizing 3D-TV, the experience of interactively watching real-world dynamic scenes from arbitrary perspective, are currently being put out of the way by researchers all over the world.

The 3D TV concept was put forward in the ATTEST project, which started in March 2002 as part of the Information Society Technologies (IST) programme, sponsored by the European Commission. Here, several industrial and academic partners cooperate towards a flexible, 2D-compatible and commercially feasible 3D TV system for broadcast environments.

An entire 3D-video chain was developed. The goals were content creation, coding, transmission, display and the central role that human 3D perception research will play in optimizing the entire chain. The goals include the development of a new 3D camera, algorithms to convert existing 2D-video material into 3D, a 2D-compatible coding and transmission scheme for 3D video using MPEG-2/4/7, and two new autostereoscopic displays. Obviously, if a workable and commercially acceptable solution can be found, the introduction of 3D TV will generate a huge replacement market for the current 2D-TV sets. In this decade, we expect that technology will have progressed far enough to make a full  3D-TV application available to the mass consumer market, including content    generation, coding, transmission and display.

1. INTRODUCTION

3D TV is expected to be the next major revolution in the history of television. Both at professional and consumer electronics exhibitions, 3D-video and 3D displays always attract a lot of interest. Obviously, if a workable and commercially acceptable solution can be found, the introduction of 3D-TV will generate a huge replacement market for the current 2D-TV sets.

This paper describes the ATTEST project that started in March 2002 as part of the Information Society Technologies (IST) programme , sponsored by the European Commission. In the 2-year project, several industrial and academic partners cooperate towards a flexible, 2D-compatible and commercially feasible 3DTV system for broadcast environments. 3D TV includes real-time acquisition, coding, and transmission of dynamic scenes.

The entire 3D-video chain block diagram is shown in Fig.1.We use an array of hardware-synchronized cameras to capture multiple perspective views of the scene. The need for the 3D-video content will be satisfied in two different ways. First, a range camera will be converted into a broadcast 3D camera that will require a redesign of the camera optics and electronics. Secondly, as the need for 3D content can only partially be satisfied by newly recorded material, a new algorithm will also be developed to convert existing 2D-video material into 3D. Both offline (content provider) and online (set-top box) conversion tools will be provided. Compatibility with conventional 2D-TV is of vital importance, as 2D and 3D-TV will co-exist during the evolution period. Therefore, the coding schemes will be developed within the current MPEG-2/4/7 broadcast standards that allow for the transmission of depth information in an enhancement layer, while providing full compatibility with existing 2D decoders. For transmission, a DVB network will be used. As the area of 3D displays is still rapidly evolving, the video chain should be adaptable to a wide range of both 2D and 3D displays. The transmitted video plus depth information allows for rendering images for many such displays. Two new 3D displays will be developed, one for a single user and one for multiple users. Both allow free viewing (without stereo-glasses) over a wide viewing angle. The 3D display provides horizontal parallax with 16 independent perspective views at 1024×768 resolution. Head tracking will be used to drive the display optics such that the appropriate images are projected into the eyes of the viewers.

These three-dimensional displays hold tremendous potential for many applications in entertainment, information presentation, reconnaissance, tele-presence, medicine, visualization, remote manipulation and art.

In the following sections we will elaborate in greater detail on the individual parts of the 3D-video processing chain. We will start with a brief description of the content generation part, where we will discuss the development of the novel 3D camera as well as the 2D-to-3D conversion. Special emphasis will then be put on the coding and transmission aspects. This will be followed by some results for the novel view generation, a short description of the 3D displays and the human perceptual evaluation of the 3D-video chain. We will finish the paper with a short conclusion.

2. SUBJECT DETAILING

2.1 CONTENT CREATION

The 3D-video content will be supplied by novel 3D cameras and via conversion from existing 2D-video material .The content creation depends upon whether the object is static or dynamic.

2.1.1 NOVEL CAMERAS FOR 3D STATIC VIDEO

The 3D-video camera that will be developed is based on Zcam™; an existing depth camera. In the project, the camera will be improved to meet the resolution and accuracy demands of 3D-TV. Next we compare the new approach with more conventional approaches using multiple 2D cameras, followed by a discussion on the used technology and its challenges.

2.1.1. a DEPTH CAMERA VERSUS MULTIPLE 2D CAMERAS

The new depth camera will yield conventional 2D-video accompanied with depth-per-pixel via a direct depth sensing process.In the case of conventional approaches with a stereo camera consisting of two or more conventional 2D cameras the actual 3D-video is then acquired via subsequent image processing such as camera calibration, correspondence estimation and stereo triangulation.

However, the accuracy of stereo triangulation deteriorates with the distance from the cameras. As video production requires the ability to shoot both close-ups as well as long distance shots, the stereo accuracy will thus vary accordingly. The changing production demands require also changing the camera geometry, e.g. zooming via precision setting of two or more independent lenses. This is mechanically impractical and requires reliable calibration tools. Finally, correspondence estimation depends on the existence of matchable features in the scene content and is still prone to errors due to mismatches. A scene point must be visible by at least two cameras if its depth is to be recovered.

Since the cameras have different positions, it is common that there are areas that are visible by a single camera only. This problem is reduced by increasing the number of 2D cameras, but this will make the stereo camera setup more difficult to handle during production and increase the amount of video streams to be processed.

The camera developed in the ATTEST project overcomes the aforementioned obstacles. The depth camera is based on a single sensor that measures the distance from the camera to the scene at each pixel simultaneously. There are no angular differences between the color camera and the depth sensor, so each pixel of the color camera is assigned a corresponding depth value. The depth measurement is independent of the visible scene content: a depth map is generated even if the scene contains no visible features (e.g. in total darkness). This assures correct recovery of depth maps for areas of constant color, for example cloths and walls. The depth accuracy of the camera is independent of the distance from the camera. The camera measures linearly scaled depth values inside a controllable depth range. This enables the camera to handle seamless changes between long distance shots and close-ups, without affecting the quality of the recovered depth and without any change of the camera geometry (such as a change of base line).

2.1.1.b THE TECHNOLOGY OF THE DEPTH CAMERA

The operation of the camera is based on generating a “light wall” moving along the field of view, see Figure 2.1.1.a. As the light wall hits the objects, it is reflected back towards the camera carrying an imprint of the objects. The imprint contains all the information required for the construction of the depth map. The 3D information can now be extracted from the reflected deformed “wall” by deploying a fast image shutter in front of the CCD chip and blocking the incoming light as shown in Figure 2.1.1.a.c. This type of camera belongs to a broader group of sensors known as scanner-less LIDAR (laser radar without mechanical scanner). The collected light at each of the pixels is related to depth, but also to the reflectivity of objects. Hence, a normalization step is performed per pixel by dividing the front portion pixel intensity by the corresponding portion of the portion of the total intensity.

Figure 2.1.1.a : (a) Light wall moving from camera to scene (b) Imprinted light wall back to camera (c) Truncated light wall containing depth information from the source.

The output of the depth camera will be of two streams .One will be the normal broadcast quality image and the other will be the depth image. Figure 2.1.1.b and Fig. 2.1.1.c shows the video and depth images taken by the current camera.

Figure 2.1.1.b & Figure 2.1.1.c : Images taken by the depth camera; (a) normal RGB image;(b) accompanying depth image ( graylevel inversely proportional to depth)

The technological challenge of the depth camera is two fold :

1. Fast switching of the illumination source to form the “light wall”, and fast gating of the reflected image entering the camera. In the current depth camera, a cluster of IR laser diodes and corresponding optics is used to generate homogeneous illumination. The diodes are switched on and off with rise/fall times shorter then 1 nsec. Here, super fast driver electronics are designed to comply with the fast response, small space and low cost, and yet maintain high efficiency

2. The detection of the reflected pulse has to be synchronous with the switched illuminator. For this, a special fast driver has been designed that has rise/fall times shorter then 1 nsec. The current camera uses a fast optical switch on the basis of a so-called gated intensifier. This device is pixelized and contributes a small amount of noise, which limit the depth resolution and accuracy respectively. In the project, we will develop a solid-state shutter, which circumvents both limitations.

2.1.2 CREATION OF A 3D DYNAMIC VIDEO

Using an array of hardware-synchronized cameras captures the dynamic scenes. There are many studios where 3D dynamic videos are captured .Some of them are the virtualized reality system of Kanade et al. with 51 cameras arranged in a geodesic dome.

The Blue-C system at ETH Z¨urich consists of a room-sized environment with real-time capture and spatially-immersive display. The Argus research project of the Air Force uses 64 cameras that are arranged in a large semi-circle.

In the MetaVision project shown in Fig.2.1.2.a three cameras were used which will capture three video signals, one from the main high- resolution colour camera, and two from lower resolution auxiliary cameras positioned either side of the main camera. The auxiliary cameras are small monochrome cameras with normal TV resolution, whereas the main camera will have HDTV resolution or beyond. The auxiliary cameras are synchronized with the main camera. All the cameras are at 25Hz progressive.

Figure 2.1.2.a : Multiple camera setup in MetaVision Project

The ATTEST project described consists of an array of 16 hardware synchronized cameras as shown in Fig.2.1.2.b. Each camera captures progressive high-definition video in real-time. Here 16 Basler A101fc color cameras with 1300×1030, 8 bits per pixel CCD sensors. The cameras are connected by IEEE-1394 (FireWire) High Performance Serial Bus to the producer PCs. The maximum transmitted frame rate at full resolution is 12 frames per second. Two cameras each are connected to one of eight producer PCs.

All PCs in the prototype should have 3 GHz Pentium 4 processors, 2 GB of RAM, and run Windows XP. The Basler camera is primarily chosen because it has an external trigger that allows for complete control over the video timing. A PCI card is used with custom programmable logic devices (CPLD) that generates the synchronization signal for all cameras. All 16 cameras are individually connected to the card, which is plugged into one of the producer PCs. Although it would be possible to use software synchronization, precise hardware synchronization is preferred for dynamic scenes. The 16 cameras in a arranged in a regularly spaced linear array as shown in Figure 2.1.2.b

Figure 2.1.2.b : An array of 16 cameras and projectors used in ATTEST project.

The optical axis of each camera is roughly perpendicular to a common camera plane. Standard calibration procedures are used to determine the intrinsic and extrinsic camera parameters.

2.1.3 CONVERSION FROM CONVENTIONAL 2D VIDEO

The need for 3D-video content can only partially be satisfied with newly recorded material. Therefore, depth reconstruction algorithms will be developed that can be used to convert existing 2D-video material into 3D.

There are two types of conversion methods :

1. Off-line conversion tools will be provided for use at the broadcaster and content provider side. These will primarily be used to convert popular movies and impressive documentaries. As there aren’t any real-time constraints in this case, all available video data can be used in the computations and 3D information can be integrated over a whole shoot, resulting in high quality 3D reconstructions.

In this case computations can be performed off-line and computationally more expensive algorithms can be used. Another important advantage is that all image data is available at once. If necessary, 3D information obtained in one camera shot can even be used to provide depth augmentation for another.

2. On-line conversion methods (e.g. for processing in a set-top box), that will allow the viewer to augment any suitable, incoming 2D broadcast to 3D. In this case, computations can only be based on video frames that have already been received. The first approach will enable on-line depth augmentation using a set-top-box at the receiver end. This allows a user at home to activate 3D depth augmentation for any suitable 2D-video content that is received. In this case, computations can only be based on video frames that have already been received. Real time implementations like these require that the developed approach can take full advantage of advanced DSP capabilities.

2.2 COMPRESSION

Multi-video recordings constitute a huge amount of raw image data. Transmitting 16 uncompressed video streams with 1300×1030 resolution and 24 bits per pixel at 30 frames per second requires 14.4 Gb/sec bandwidth, which is well beyond current broadcast capabilities. So compression techniques should be adapted.

The different methods for compression are :

1. Spatial encoding

2. Temporal encoding

3. Centralized processor encoding.

Motion compensation in the time domain is called temporal encoding, and disparity prediction between cameras is called spatial encoding. Results show that a combination of temporal and spatial encoding leads to good results. The Blue-C system converts the multiview video into 3D “video fragments” that are then compressed and transmitted. However, all current systems use a centralized processor for compression, which limits their scalability in the number of compressed views.

To efficiently encode the multi-video data using object geometry, the images may be regarded as object textures. In the texture domain, a point on the object surface has fixed coordinates, and its color (texture) varies only due to illumination changes and/or non-Lambertian reflectance characteristics. For model-based coding, a texture parameterization is first constructed for the geometry model. Having transformed all multi-video frames to textures,the multi-view textures are then processed to de-correlate them with respect to temporal evolution as well as viewing direction. Shape-adaptive as well as multi-dimensional wavelet coding schemes lend themselves to efficient, progressive compression of texture information. Temporarily invisible texture regions can be interpolated from previous and/or future textures, and generic texture information can be used to fill in regions that have not been recorded at all. For spacetime isosurface reconstruction, deriving one common texture parameterization for all time instants is not trivial since the reconstruction algorithm does not provide surface correspondences over time. Encoding the time-varying geometry is also more complex than in the case of model- based analysis. Current research therefore focuses on additionally retrieving correspondence information during isosurface reconstruction.

The approach specified in the ATTEST project is to reduce the data to a single view with per-pixel depth map. This data can be compressed in real-time and broadcast as an MPEG-2 enhancement layer. On the receiver side, stereo or multiview images are generated using image-based rendering. However, it may be difficult to generate high-quality output because of occlusions or high disparity in the scene. Moreover, a single view cannot capture view-dependent appearance effects, such as reflections and specular highlights. High-quality 3D TV broadcasting requires that all the views are transmitted to multiple users simultaneously. The 3D TV system described uses temporal compression only and transmits all of the views as independent MPEG-2 video streams.

2.3 CODING AND TRANSMISSION

The 3D TV system coding should based on a flexible, modular and open architecture that provides important system features, such as backwards compatibility to today’s 2D digital TV, scalability in terms of receiver complexity and adaptability to a wide range of different 2D and 3D displays .

The 3D TV uses the Layered Coding Syntax shown in Fig.2.3.1.

Figure 2.3.1 : Layered Coding Syntax

The Layered Coding Syntax consists of one base layer and at least one additional enhancement layer. To achieve backwards compatibility to today’s conventional 2D digital TV, the base layer is encoded by using state-of-the-art MPEG-2 and DVB standards. Thus, this layer can be decoded by standard set-top boxes designed for 2D digital TV broadcast reception. The remaining enhancement layer(s) deliver(s) the additional information to the 3D-TV receiver. The minimum information transmitted in the enhancement layer(s) is an associated depth map providing one depth value for each pixel of the base layer. However, in the case of critical video content (e.g. large scale scenes with a high amount occlusions) it might be useful to send further information, for example segmentation masks and occluded texture.Note that the layered structure in Fig. 3 is extendable in this sense.

For the transmission of the enhancement layer(s), it is planed to rely as far as possible on already available MPEG-2/4/7 tools. In case that existing tools prove to be inadequate coding will be given to appropriate standardization bodies like MPEG Ad-hoc group (AHG) etc

Additionally it is important to realize that stereovision is only one of the relevant depth cues and that other cues such as motion-parallax, texture, brightness and geometric appearance of video objects are of comparable importance .

For scene objects that are sufficiently far away from the viewer, they can even become dominant see Fig.2.3.2.

Figure 2.3.2 : Importance of depth cues in dependence of the viewing distance

It is therefore a significant feature of the described layered structure that it is flexible enough to support alternative forms of depth representation. This allows for a stepwise introduction of 3D-TV receivers of different complexity. For example, an intermediate low-cost 3D-TV receiver could use the additional depth layer to render individual perspective views according to the head-tracked viewing position of the TV watcher (see also Fig. 2.3.1). By this means, broadcasters could provide the user with a first, limited depth impression through parallax viewing, even on conventional 2D-TV screens. On the other hand side, users willing to invest into a 3D-TV set could enjoy a full-blown stereo reproduction of the same data on single- or even multiple user 3D displays (see Fig. 2.3.1).

The proposed layered coding syntax will also provide scalability in terms of depth experience. This is particularly important, as perception studies have indicated that there are differences in depth appreciation over age groups. Hence in our view, the TV viewer should be in control of his depth experience. He should be able to set the depth level according to his personal preference – a feature which can also be used for graceful degradation in the case of unexpected artifacts in depth which are usually more annoying in stereovision than in parallax viewing.

The Layered coding syntax has many advantages like :

(a) Backward Compatibility – Since the base layer is coded by using MPEG,DVB standards this can be decoded by using the conventional 2D receivers ensuring that the existing 2D TV owners can also watch the transmission of 3D in 2D.Thus the system can plug into today’s digital TV broadcast infrastructure and co-exist in perfect harmony with 2D TV.

(b) Scalability-The system is highly scalable in terms of depth experience. This is particularly important, as perception studies have indicated that there are differences in depth appreciation over age groups. Hence, the TV viewer should be in control of his depth experience. He should be able to set the depth level according to his personal preference – a feature which can also be used for graceful degradation in the case of unexpected artifacts in depth which are usually more annoying in stereovision than in parallax viewing.

(c) Adaptability-The system is adaptable as there are different enhancement layers and the user can use receivers of different complexity according to the depth requirement.

(d) Another advantage of using 2D coding standards is that the codecs are well established and widely available. Tomorrow’s digital TV set-top box could contain one or many decoders, depending whether the display is 2D or multiview 3D capable.

2.4 3D DISPLAYS

The displays plays an important role in the 3D TV system. There are different types displays that can be used to produce the 3D effect.

2.4.1 HOLOGRAPHIC DISPLAYS

In holographic reproduction, light from an illumination source is diffracted by interference fringes on the holographic surface to reconstruct the light wavefront of the original object. A hologram displays a continuous analog lightfield, and real-time acquisition and display of holograms has long been considered the “holy grail” of 3D TV. The most recent device, the Mark-II Holographic Video Display, uses acousto-optic modulators, beam splitters, moving mirrors, and lenses to create interactive holograms. In more recent systems, moving parts have been eliminated by replacing the acousto-optic modulators with LCD , focused light arrays , optically-addressed spatial modulators or digital micro mirror devices. All current holo-video devices use single-color laser light. To reduce the amount of display data they provide only horizontal parallax. The display hardware is very large in relation to the size of the image (which is typically a few millimeters in each dimension). The acquisition of holograms still demands carefully controlled physical processes and cannot be done in real-time. Holographic systems are unable to acquire, transmit, and display dynamic, natural scenes on large displays.

2.4.2 VOLUMETRIC DISPLAYS

Volumetric displays use a medium to fill or scan a three-dimensional space and individually address and illuminate small voxels. However, volumetric systems produce transparent images that do not provide a fully convincing three-dimensional experience. Furthermore, they cannot correctly reproduce the lightfield of a natural scene because of their limited color reproduction and lack of occlusions. The design of large-size volumetric displays also poses some difficult obstacles that in maintaining view-dependent effects such as occlusion, specularity, and reflection. Their prototype uses beam-splitters to emit light at focal planes at different physical distances. Two such devices are needed for stereo viewing. Since the head and viewing positions remain fixed, this prototype is not a practical 3D display solution.

2.4.3 PARALLAX DISPLAYS

Parallax displays emit spatially varying directional light. They provide only horizontal parallax . There are different types of parallax displays:

(a) Parallax Stereograms – It uses a plate with vertical slits as a barrier over an image with alternating strips of left-eye/right-eye images .

(b) Parallax Panoramagrams – To extend the limited viewing angle and restricted viewing position of stereograms, narrower slits and smaller pitch are used between the alternating image stripes .

(c) Integral photograph – It uses an array of spherical lenses instead of slits. This is frequently called a “fly seye” lens sheet. An integral photograph is a true planar lightfield with directionally varying radiance per pixel . Integral lens sheets can be put on top of high-resolution LCDs. Integral photographs sacrifice significant spatial resolution in both dimensions to gain full parallax.

2.4.4 LENTICULAR DISPLAYS

The most common display material used is the Lenticular sheet. Lenticular sheet is a linear array of narrow cylindrical lenses called Lenticules. This reduces the amount of image data by giving up vertical parallax. To improve the native resolution of the display, multi – projector lenticular displays are used.. For this the back of a lenticular sheet is painted with a diffuse paint and is used as a projection surface. Different arrangements of lenticular sheets and multi-projector arrays can be made.

For the 3D display the lenticules are arranged as either a Rear-projection or a Front-projection 3D display. The display system uses a linear array of 16 projectors and lenticular screens.

The system uses 16 NEC LT-170 projectors with 1024×768 native output resolution. This projector is chosen because of its compact form factor which proposes values for optimal projector separation and lens pitch. Ideally, the separations between cameras and projectors are equal. The offset in the vertical direction between neighboring projectors leads to a slight loss of vertical resolution in the final image. The system uses eight consumer PCs and dedicate one of them as the controller.

For the rear-projection system (Figure 2.4.4.a left), two lenticular sheets are mounted back-to-back with optical diffuser material in the center. The back-to-back lenticular sheets and the diffuser fabric were composited using transparent resin that was UV-hardened after hand-alignment.

The front-projection system (Figure 2.4.4.a right) uses only one lenticular sheet with a retro-reflective front-projection screen material mounted on the back.

Figure 2.4.4.a : Projection type Lenticular 3D display

The projection-side lenticular sheet of the rear-projection display acts as a light multiplexer, focusing the projected light as thin vertical stripes onto the diffuser. A closeup of the lenticular sheet is shown in Figure 2.4.4.b. Considering each lenticule to be an ideal pinhole camera, the stripes capture the view-dependent radiance of a three-dimensional lightfield (2D position and azimuth angle). The viewer-side lenticular sheet acts as a light de-multiplexer and projects the view-dependent radiance back to the viewer. Note that the single lenticular sheet of the front-projection screen both multiplexes and de-multiplexes the light.

Figure 2.4.4.b : Formation of vertical stripes on the diffuser of the Rear-projection display. The closeup photograph shows the lenticules and stripes from one view point.

Figure 2.4.4.c shows photographs of both rear-projection and front-projection displays.

Figure 2.4.4 c : Rear-projection 3D display with double-lenticular screen. Right: Front-projection 3D display with single- lenticular screen.

The two key parameters of lenticular sheets are the field of view (FOV) and the number of lenticules per inch (LPI).The lenticular sheets used here are of 72”× 48” size with 30 degrees FOV and 15 LPI. The optical design of the lenticules is optimized for multiview 3D display. The number of viewing zones of a lenticular display are related to its FOV (see Figure 2.4.4.d). Here, the FOV is 30 degrees, leading to 180/30 = 6 viewing zones.

At the border between two neighboring viewing zones there is an abrupt view-image change (or “jump”) from view number 16 to view number one. This is eliminated by increasing the FOV of the display. Here each sub pixel (or thin vertical stripe) in Figure 7 is projected from a different projector, and each projector displays images from a different view.

The field of view of a lenticular display.

2.4.4.1 DISPLAY CALIBRATION

Automatic projector calibration for the 3D display is very important. Here relationship between rays in space and pixels in the projected images is founded by placing a camera on the projection side of the screen. Then the intensities of the projectors are equalized. For both processes, the display is covered with a diffuse screen material. Standard computer vision techniques are used to find the mapping of points on the display to camera pixels, which can be expressed by a 3×3 homography matrix. The largest common display area is computed by fitting the largest rectangle of a given aspect ratio (e.g., 4:3) into the intersection of all projected images. Even for one projector, the intensities observed by the camera vary throughout the projected image. Moreover, different projectors may project images of vastly different intensities.

The calibration procedure works as follows. First, a white image is projected into the common rectangle plane with each projector. Then record the minimum intensity in this image for each projector and then determine the minimum of those values across all projectors. This is the maximum intensity that is used for equalization.

Next, iteratively adjust the intensity of the image for each projector until the observed image has even maximum intensity.

This is possible because we know the correspondence between the camera pixels and the pixels of each projector. This process yields image-intensity masks for each projector. It is only an approximate solution, ince the response of the projectors for different intensities is generally nonlinear. In the rear-projection system, a translation of the lenticular sheets with respect to each other leads to an apparent rotation of the viewing zones (see Figure 2.4.4.e).

Figure 2.4.4.e : Apparent viewing zone rotation for rear-projection due to a shift of the lenticular screens.

In order to estimate this horizontal shift turn on each of the 16 projectors separately and measure the response on the viewer side of the display using the camera. The camera is placed approximately in the center of the display at the same distance to the screen as the projectors. Then observe with which projector maximum brightness in the camera is achieved. This is called as the apparent central projector. The image data is then re-routed between decoders (producers) and consumers such that the apparent central projector receives the images of the central camera.

2.5 PERCEPTUAL EVALUATION

The acceptance, uptake, and commercial success of any advanced technology aimed at the consumer market depend to a large extent on the users experiences with and responses towards the system. In the past, 3D video in theme parks, and even in 3D broadcast trials, were often intended to provide the viewers the ‘3D thrill of their life’. Depth impressions were also exaggerated to enhance the visual impact. Unfortunately, viewers also frequently experienced eye strain, headaches, and other unpleasant side effects. Therefore, it is vital to have a clear understanding of the in-the-home viewing experience of 3D-TV, both looking at the potential added value of the ATTEST 3D-TV systems, as well as the potential drawbacks for users. Our aim is to arrive at a set of requirements and recommendations for an optimal 3D-TV system, and contribute to each individual step in the 3D video chain through perceptual and usability evaluations of the proposed technological innovations.

More specifically, human-factors experiments will be performed to address the depth impression, perception of distortions, eye strain, quality, naturalness, presence, and acceptability of the 3D coding algorithms and novel 3D displays, in order to arrive at perceptually optimal image quality with minimal coding artifacts and negligible side-effects . Additionally, a number of basic and novel areas surrounding 3D video perception will be investigated that will enhance our understanding of the user experience. For example, user control over the depth impression in 3D video has to date received very little systematic experimental investigation. This will be one of the issues that will be addressed, looking at both basic perceptual and cognitive effects as well as ease-of-use. In addition, the fundamental issue of acceptability of 2D production grammars for 3D video will be investigated, requiring a much deeper understanding of how depth perception develops over time – e.g., how tolerant viewers will be to sudden disparity changes – whilst relating these insights to existing 2D and 3D video production grammars.

3. ADVANTAGES AND DISADVANTEGES

Like any other system the 3D TV system has both advantages and disadvantages.

3.1 ADVANTAGES

1. The 3D TV display shows high resolution of 1024 X 768 pixels of stereoscopic color images for multiple viewpoints without special glasses.

2. The system is completely scalable and backwad compatible in the number of acquired, transmitted and displayed views.

3. The new algorithm efficiently renders novel views from multiple dynamic video streams on a cluster of PC’s.

4. The large number of views (16) , and the large physical dimension

( 6’X4’ ) of the display lead to a very immersive 3D experience.

5. The projector based 3D display has a native resolution of 12 million pixels which is greater than the largest currently available high resolution flat-panel screen of IBM T221 LCD with 9 million pixels.

6. The overall delay in the system fron the acquisition to the display is less than one second.

3.2 DISADVANTAGES

Some of the disadvantage of the 3D TV system are:

1. The graphics cards and projectors are not synchronized which lead to and increased motion blur for fast movements in the scene.

2. The Rear projection system has less quality compared to front projection system as it exhibits Moir’e artifacts on the screen.

3. The front projection system has more difficulty to represent pure blacks, or color when the variations between the projected images are more apparent.

4. The lenticular sheet with 15 LPI shows some visible vertical lines, which will vanish when the number of lenticules per inch are increased.

4. APPLICATIONS AND FUTURE WORK

4.1 APPLICATIONS

The 3D TV has a number of applications. Some of them are :

1. Applications in film and TV production : The 3D technology is used in film and TV production for capturing 3D information from image sequences The potential applications fall into two classes, one requiring 3D data that can be represented as a depth map from a single viewpoint, and the other requiring a full 3D model. Applications for both classes of data are briefly reviewed, and current work on 3D data capture in two EU-funded projects are the MetaVision project which is considering depth map acquisition based on a three-camera stereo system. The development of a multi-camera system using widely separated cameras in a studio environment is being carried out as a part of the ORIGAMI project.

2. Tele-conferencing : Since the delay between the acquisition and display is less than one second ,the system can be used for tele-conferencing purposes if suitable multiview video compression techniques are available in future.

3. Medical field : 3D displays can be used in the medical field for the effective diagnosis applications.

Figure 4.1. Visualization of different structures of the Visible Male knee on the 3D-LCD Philips prototype monitor.

4.Entertainment : The 3D TV system by its resolution of 1300 X 768 pixels will provide the TV viewers with the most natural viewing experience .It will provide the experience of seeing through the window .

4.2 FUTURE WORK

Most of the key ideas for the 3D TV system presented in this paper have been known for decades, such as lenticular screens, multiprojector 3D displays, and camera arrays for acquisition. The system is the first to provide enough viewpoints and enough pixels per viewpoint to produce an immersive and convincing 3D experience without special glasses. It is also the first system that provides this experience in real-time for dynamic scenes.

There is still much that we can do to improve the quality of the 3D display. As noted before, the rear-projection system exhibits Moir´e artifacts that can only be corrected by very precise vertical alignment of the lenticular sheets. The type of screen material (diffuse or retro-reflective) has a huge influence on the quality and sharpness of either rear- or front-projection screens. Experiments will be done with different lenticular sheets to improve the FOV and sharpness of the display. To improve the optical characteristics of the 3D display computationally. We call this concept the computational display. First, we plan to estimate the light transport matrix (LTM) of our view-dependent display by projecting patterns and observing them with a camera array on the viewer side. Knowing the LTM of the display will then allow us to modify the projected images to improve the quality. The viewing-side cameras could also be replaced by a user who can tune the display parameters using a remote control to find the best viewing condition for the current viewpoint. Another area of future research is precise color reproduction of natural scenes on multiview displays. Another new and exciting area of research is high-dynamic range 3D TV. High-dynamic range cameras are being developed commercially and have been simulated using stereo cameras . True high-dynamic range displays have also been developed . We plan to extend these methods to multiview camera arrays and 3D displays. We also plan to experiment with multiview displays for deformable display media, such as organic LEDs. Multiview cameras and displays that dynamically change their parameters, such as position, orientation, focus, or aperture, pose new and exciting research problems for the future.

5. CONCLUSION

It is impossible to convey the impression of dynamic 3D TV on paper. The 3D-TV system will be an entire 3D-video chain including content creation, coding, transmission and display. All parts will be optimized with respect to the entire chain, guided by research on human 3D perception.

We discussed the specific goals for all system parts. A new 3D camera will be developed that meets the resolution and accuracy requirements of the 3D-TV application. Both real-time and off-line algorithms will be developed to convert existing 2D-video material into 3D. For transmission, we use a 2D-compatible method in which conventional images are accompanied with depth information, coded with MPEG-2/4/7 schemes. This scheme enables addressing of a wide range of 2D and 3D displays. Finally, two autostereoscopic displays will be developed ;one optimized for a single viewer, and a second display for multiple viewers.

With the combination of well-established academic and industrial partners, and building upon the technological progress obtained from earlier 3D projects, we expect to achieve the goal of developing the first commercially feasible 3-dimensional television broadcast system which will provide the most natural viewing experience.

1. INTRODUCTION

There is a saying in real estate; when land get expensive, multi-storied buildings are the alternative solution. We have a similar situation in the chip industry. For the past thirty years, chip designers have considered whether building integrated circuits multiple layers might create cheaper, more powerful chips.

Performance of deep-sub micrometer very large scale integrated (VLSI) circuits is being increasingly dominated by the interconnects due to increasing wire pitch and increasing die size. Additionally, heterogeneous integration of different technologies on one single chip is becoming increasingly desirable, for which planar (2-D) ICs may not be suitable.

The three dimensional (3-D) chip design strategy exploits the vertical dimension to alleviate the interconnect related problems and to facilitate heterogeneous integration of technologies to realize system on a chip (SoC) design.  By simply dividing a planar chip into separate blocks, each occupying a separate physical level interconnected by short and vertical interlayer interconnects (VILICs), significant improvement in performance and reduction in wire-limited chip area can be achieved.

In the 3-Ddesign architecture, an entire chip is divided into a number of blocks, and each block is placed on a separate layer of Si that are stacked on top of each other.

2. MOTIVATION FOR 3-D ICs

The unprecedented   growth of   the computer and the information technology industry is demanding Very Large Scale Integrated ( VLSI ) circuits with increasing functionality and performance at minimum cost and power dissipation. Continuous scaling of VLSI circuits is reducing gate delays but rapidly increasing interconnect delays. A significant fraction of the total power consumption can be due to the wiring network used for clock distribution,  which is usually realized using long global wires.

Furthermore, increasing drive for the integration of disparate signals (digital, analog, RF) and technologies (SOI, SiGe, GaAs, and so on) is introducing various SoC design concepts, for which existing planner (2-D) IC design may not be suitable.

INTERCONNECT LIMITED VLSI PERFORMANCE

In single Si layer (2-D) ICs, chip size is continuously increasing despite reductions in feature size made possible by advances in IC technology such as lithography and etching. This is due to the ever growing demand for functionality and high performance, which causes increased complexity of chip design, requiring more and more transistors to be closely packed and connected. Small feature sizes have dramatically improved device performance. The impact of this miniaturization on the performance of interconnect wire, however, has been less positive. Smaller wire cross sections, smaller wire pitch, and longer line to traverse larger chips have increase the resistance and capacitance of these lines, resulting in a significant increase in signal propagation (RC) delay. As interconnect scaling continues, RC delay is increasingly becoming the dominant factor determining the performance of advanced IC’s.

PHYSICAL LIMITATIONS OF Cu INTERCONNECTS

At 250 nm technology node, Cu with low-k dielectric was introduced to alleviate the adverse effect of increasing interconnect delay.However,below 130nm technology node, substantial interconnect delays would result in spite of introducing these new materials, which in turn will severely limit the chip performance. Further reduction in interconnect delay is not possible.

This problem is especially acute for global interconnects, which comprise about 10% of total wiring in current architectures.  Therefore, it is apparent that material limitations will ultimately limit the performance improvement as technology scales.  Also, the problem of long lossy lines cannot be fixed by simply widening the metal lines and by using thicker interlayer dielectric, since this will leas to an increase in the number of metal layers.  This will result in an increase in complexity, reliability and cost.

SYSTEM – ON – A – CHIP DESIGN

System – on – a –chip (SoC) is a broad concept that refers to the integration of nearly all aspects of a system design on a single chip.  These chips are often mixed-signal and/or mixed-technology designs, including such diverse combinations as embedded DRAM, high – performance and low-power logic, analog, RF, programmable platforms (software, FPGAs, Flash, etc.).

SoC designs are often driven by the ever-growing demand for increased system functionality and compactness at minimum cost, power consumption, and time to market. These designs form the basis for numerous novel electronic applications in the near future, in areas such as wired and wireless multimedia communications including high speed internet applications, medical applications including remote surgery, automated drug delivery, and non invasive internal scanning and diagnosis, aircraft/automobile control and safety, fully automated industrial control systems, chemical and biological hazard detection, and home security and entertainment systems, to name a few.

There are several challenges to effective SoC designs:

1.      Large scale integration of functionalities and disparate technologies on a single chip  dramatically increases the chip area, which necessitates the use of numerous long global wires.  These wires can lead to unacceptable signal transmission delays and increase the  power consumption by increasing the total capacitance that needs to be driven by the gates.

2.      Integration of disparate technologies such as embedded DRAM, logic, and passive components in SoC applications introduces significant complexity in materials and process integration.

3.      The noise generated by the interference between different embedded circuit blocks containing digital and analog circuits becomes a challenging problem.

4.      Although SoC designs typically reduce the number of  I/O pins compared to a system assembled on a printed circuit board(PCB), several high performance SoC designs involve very high I/O pin counts , which can increase the cost per chip

5.      Integration of mixed technologies on a single die requires novel design methodologies and tools ,with design productivity being a key requirement.

Ø 3D ARCHITECTURE

Three-dimensional integration to create multilayer Si ICs is a concept that can significantly improve interconnect performance ,increase transistor packing density, and reduce chip area and power dissipation. Additionally 3D ICs can be very effective large scale on chip integration of different systems.

In 3D design architecture, and entire(2D) chips is divided into a number of blocks is placed on separate layer of Si that are stacked on top of each other. Each Si layer in the 3D structure can have multiple layer of interconnects(VILICs) and common global interconnects.

ADVANTAGES OF 3D ARCHITECTURE

The 3D architecture offers extra flexibility in system design, placement and routing. For instance, logic gates on a critical path can be placed very close to each other using multiple active layers. This would result in a significant reduction in RC delay  and can greatly enhance the performance of logical circuits.

• The 3D chip design technology can be exploited to build SoCs by placing circuits with different voltage  and  performance requirements in different layers.
• The 3D integration can reduce the wiring ,thereby reducing the capacitance, power dissipation and chip area and therefore improve chip  performance.
• Additionally the digital and analog components in the mixed-signal systems can be placed on different Si layers thereby achieving better noise performance due to lower electromagnetic interference between such circuits blocks.
• From an integration point of  view, mixed-technology assimilation could be made less complex and more cost effective by fabricating such technologies on separate substrates followed by physical bonding.

3. SCOPE OF THIS STUDY

A 3D solution  at first glance seems an obvious answer to the interconnect delay problem. Since chip size directly affects the inter connect delay, therefore by creating a second active layer, the total chip footprint can be reduced, thus shortening critical inter connects and reducing their delay. However, in today’s microprocessor, the chip size is not just limited by the cell size ,but also by how much meta is required to connect the cells.  The transistors on the Si surface are not actually packed to maximum density, but  are spaced apart to allow metal lines above  to connect one transistor or one cell to another .The meal required on a chip for inter connections is determined  not only by the number of gates ,but also by other factors such as architecture, average fan-out, number of I/O connections, routing complexity, etc Therefore, it is not obvious that using a 3D structure the chip size will be reduced.

4. AREA AND PERFORMANCE ESTIMATION OF 3D ICs

Now we present a methodology that can be used to provide an initial estimate of the area and performance of high speed logic circuits fabricated using multiple silicon layer IC technology. The approach is based on the empirical relationship known as Rent’s Rule.

Rent’s Rule:

It correlates the number of signal input and output (I/O) pins T, to the number of gates N, in a random logic network and is given by the following expressions :

T=kN^P ————-(i)

Here k & P denote the average number of fan out per gate and the degree of wiring complexity (with P=1 representing the most complex wiring network), respectively, and are empirically derived as constants for a given generation of ICs.

A) 2-D AND 3-D WIRE-LENGTH DISTRIBUTIONS

The wire-length distribution can be described by i(l),an interconnect density functions (i.d.f), or by I(l), the cumulative interconnect distribution function (c.i.d.f) which gives the total number of interconnects that have length less than or equal to l (measured in gate pitches) and is defined as

/

I(l)= i(x)dx         ———–(ii)

Where x is a variable of integration representing length and l is the length of the interconnect in gate pitches. The derivation of the wire-length distributed in a Ic is based on Rent’s Rule. To  derive the wire length distribution I(l) of an integrated circuit, the latter is divided up into N logic gates, where N is related to the  total number of transistor Nt in an integrated circuit by N=Nt/O where O is a function of  the average fan-in(f.i0 and fan-out(f.o). The gate pitch is defined as the average separation between the logic gates and is equal to sqt(Ac/N) where Ac is the area of the chip.

In order to derive the complete wire-length distribution for a chip, the stochastic wire-length distribution of a single gate must be calculated.

The number of connections from the single logic gate in Block A to all other gate that are located at a distance of l gate pitches is determined using Rent’s Rule. The gates shown in the figure are grouped into three distinct but adjacent blocks(A,B&C), such that a closed single path can encircle one, two or three of these blocks. The number of connections between Block A and Block C is calculated by conserving all I/O terminals for blocks, A, B, and C, which states that terminals for blocks A, B, and C, are either interlock connections or external system connections.

Hence, applying the principle of conservation of I/O pins to this system of three logic blocks, shown gives

T_A + T_B + T_C =  T_{A to C} + T_{A to B} + T_{B to C} + T_ABC …………….(iii)

Where T_A, T_B, T_C are the number of I/O blocks A, B, and C respectively.  T_{A to C} , T_{A to B}, T_{B to C} are the number of I/Os between blocks A and C, blocks A and B, and between blocks B and C, respectively.  T_ABC represents the number of I/Os  for the entire system comprising of all three blocks.  From conservation of I/Os, the number of I /Os between adjacent blocks A and B, and between adjacent blocks A and B and between adjacent blocks B and C can be expressed as

T_{A to B} =  T_A + T_B - T_AB ……………………..…….(iv)

T_{B to C} =  T_B + T_C – T_BC ..………………………….(v)

Substituting (iv) and (v) into (iii) gives

T_{A to C} = T_AB + T_BC – T_B – T_ABC …………………………(vi)

Now the number of I/O pins for any single block or a group of blocks can be calculated using Rent’s Rule.  If we assume that N, N, and N are the number of gates in blocks A, B, and C, respectively, then it follows that

T_B = k (N_B)^P ………………………(vii)

T_­­­­­­­­­_­­­­_AB = k(N_A­ + N­_B)^P ……..………………..(viii)

T­_BC = k(N_B + N_C )^P ……….………………(ix)

T_ABC = k(N_A + N­_B +N_C)^P ……………………….(x)

Where N = N_A + N­_B + N_C.   Substituting (vii) – (x) into (vi) gives

T_{A to C} = k [( N_A + N_B)^P – (N_B)^P + (N_B + N_C)^P – (N_A + N_B + N_C)^P] ……..(xi)

The number of interconnects between Block A and Block C (I_{A to C}) is determined using the relation

I_{A to C} = αk (T_{A to C})

Where α is related to the average fan out (f.o.) by

α = f.o. / (1+f.o.)

Applying Rent’s Rule to all the layers, we have

T=kN_­­­^P = (T_i) – T_int = nk(N/n)^P – T_int

Here, T  is the number of I/Os for the entire design, T_{i ­}represents the number of I/O ports connecting n layers. Hence it follows that

T_int = n (1- n^P-1) k (N/n)^P and

T_ext,i = T_i – T_int/n = kn^P-1 (N/n)^P

Here, T_ext,i , is the average of I/O ports per layer.

B) ESTIMATING 2-D AND 3-D CHIP AREA

In integrated circuits that are wire-pitch limited in size, the area require by the wiring network is assumed to be much greater than the area required by the logic gates. For the purpose of minimizing silicon real estate and signal propagation delays, the wiring network is segmented into separate tiers that are physically fabricated in multiple layers.

An interconnect tier is categorized by factors such as metal line pitch and cross-section, maximum allowable signal delay and communication mode (such as intra block, or inter block).  A tier can have more than one layer of metal interconnects if necessary, and each tier or layer is connected to the rest of the wiring network and the logic gates by vertical vias.  The tier closest to the logic devices (referred to as the local tier) is normally for short distance intra block communications.

Metal lines in this tier will normally be the shortest.  They will also normally have the finest pitch.  The tier furthest away from the device layer (referred to as global tier) is responsible for long distance across chip inter block communications, clocking and power distribution.  Since this tier is populated by the longest of wires, the metal pitch is the largest to minimize signal propagation delays.  A typical modern IC interconnects architecture will define three wiring tiers: local, semi-global, and global. The semi-global tier is normally responsible for inter block communications across intermediate distances.

The area of the chip is determined by the total wiring requirement.  IN terms of gate pitch, the total area required by the interconnect wiring can be expressed as

A_required = √A_c (P_locL_{total_loc}+P_semiL_{total_semi}+P_globL_{total_glob})/N

Where,

A_c Chip area ;

N                     number of gates;

P_loc local pitch;

P_semi semi global pitch;

P_global global pitch;

L_{total_loc} total lengths of local interconnects;

L_{total_semi} total length of semi global interconnects;

L_{total_glob} total length of global interconnects;

The total interconnects length for any tier can be found by integrating the wire-length distribution within the boundaries that define the tier. Hence it follows that

L_{total_loc}= X ∫ li (l) dl

L_{total_semi}=X ∫ li (l) dl

L_{total_glob} = X ∫ li (l) dl

Where X is a correction factor that converts the point –to – point interconnect length to wiring net length (using a linear net model, X=4/(f.o. + 3)

C) TWO ACTIVE LAYER 3-D CIRCUIT PERFORMANCE

This analysis is used to compare area and delay values for 2-D and 3-D ICs. The availability of addition of silicon layers gives the designer extra flexibility in trading off area with delay. A number of different cases are discussed as follows:

1. Chip area minimization with fixed interconnect delay

Here, VILICs are assumed to consume negligible area, interconnect line width is assumed to equal half the metal pitch at all times, and the total number of metal layers for  2-D and 3-D case was conserved. A key assumption for the geometrical construction of each tier of the multilevel interconnect network is that all cross sectional dimensions are equal within that tier.

As P_semi increase from its value at the minimum A_c the semi global and global pitches increase resulting in a larger wiring requirement and thus a larger A_c. Furthermore, as P_semi increases, even longer wires can now satisfy the maximum delay requirement in the semi global tier. These results in global wires to be rerouted to the semi global tier, which in turn will require greater chip area. Under such circumstances, the semi global tier begins to dominate and determine the chip area. Conversely, as P_semi decreases from its value at the minimum A_c, the longer wires in the semi global tier no longer satisfy the maximum delay requirement of that tier and they need to be rerouted to the global tier where they can enjoy a larger pitch. The populations of wires in global tiers increases and since these wires have a large cross section they have a greater area requirement. Under such circumstances the global tier begins to dominate and determine the chip area.

The curve for the 3-D case has a minimum similar to the one obtained for the 2-Dcase.it can be observed that the minimum chip area for the 3-D case is about ≈30% smaller than that of the 2-D case. Moreover, since the total wiring requirement is reduced, the semi global tier pitch is reduced for the 3-Dchip. The significant reductions in chip area demonstrated by the 3-D results are a consequence of the fraction of wires that were converted from horizontal in 2-D to vertical VILICs in 3-D. it is assumed that the area required by VILICs is negligible.

These results demonstrate, with given assumptions, that a 3-D IC can operate at the same performance level, as measured by the longest wire delay, as its 2-D counterpart while using up about 30% less silicon real estate. However, it is possible for 3-D ICs to achieve greater performance than their 2-Dcounterparts by reducing the interconnect impedance at the price of increased chip area as discussed next.

2. Increasing Chip Area and Performance

3-D IC performance can be enhanced to exceed the performance of 2-D ICs by improving interconnect delay. This is achieved by increasing the wire pitch, which causes a reduction in the resistance. The effect of increasing p_semi and p_global on the operating frequency and A_c.

This illustrates how the optimum semi global pitch (i.e., the p_semi associated with the minimum A_c) increases to obtain higher operating frequencies. Also, as the semi global tier pitch increases, chip area and, therefore, interconnect length also increases. However, we can see that the increase in chip area still remains well below the area required for the 2-D case. The figure also helps defines a maximum – performance 3-D chip – a chip with the same area as the corresponding 2-D chip, which can be obtained by increasing the semi global pitch beyond that for the 4-GHz case.

Two scenarios are considered 1) global pitch is increased to match the global pitch for the 2-d case and 2) global pitch is increased to match the chip area (footprint) for the 2-d case. Note that the delay requirements sets a maximum values of interconnect length are given tier. Therefore, as interconnect lengths are increased, lines which exceed this maximum length criterion for that particular tier need to be rerouted on upper ties.

Beyond the maximum performance point for the 3-d chip, the performance gain becomes increasingly smaller in comparison to the decrease in performance resulting from the increase in chip area or reconnect delay. Furthermore, as the semi global wires need to be rerouted on the global tiers, which eventually leads to overcrowding of the global tier . Any further increases in the wiring density in the global tier forces a reduction in the global pitch.

D) EFFECT OF INCREASING NUMBER OF SILICON LAYERS

As the number of silicon layers increases beyond two, the assumption that all interlayer interconnects (ILICs) are vertical and consume negligible area becomes less tenable. The area used up by these horizontal ILICs can be estimated from their total length and pitch.

The decrease in interconnect delay becomes progressively smaller as the numbers of active layers increase. This is due to the fact that the area required by ILICs begins to offset any area saving due to increasing the number of active layers.

E) EFFECT OF INCREASING THE NUMBER OF METAL LAYERS

It is likely that there are local and semi global tiers associated with every active layer, and a common global tier is used . This would result in an increase in the total number of metal layers for the 3-D case. The effect of using 3-D case. The effect of using 3-D ICs with constant metal layers and the effect of employing twice the number of metal layers as in 2-D are summarized in the figure.

It  can be observed that by using twice the number of metal layers the performance of the 3-D chip can be increased by an additional amount of 35% as compared to the 3-d chip with the same total number of metal layers as in 2-d . It can be observed that for the more aggressive technologies , the decrease in interconnect delay from 2-D to 3-D case is less impressive. This indicates that more than two active layers are possibly needed for those advanced nodes. The figure also shows the impact of moving only the repeaters to the second layer Si layer . It can also be observed that for more aggressive technologies , the decrease  in interconnect delay from 2-D to 3-D case is less impressive This indicates that more than two active layers are possibly needed for those advanced nodes.

F.OPTIMIZATION OF INTERCONNECT DISTRIBUTION

In estimating chip area, the metal requirement is calculated from the obtained wire-length distribution. The total metallization requirement is appropriately divided among the available metal layers in the corresponding technology . Thus each tier , the local , the semi global and the global has three metal layers . the resulting area of most densely packed tier determines the chip area.

Consequently, higher tier are routed within a larger than required area . An optimization for this scenario is possible by rerouting some of the local wires on the semi global tier and the latter on the global , without violating the maximum allowable  Length ( or delay ) per tier. This is achieved by reducing the maximum allowed interconnect length for the local and semi global tiers. Minimum chip area will achieved when all the tiers are equally congested. The 2-D chip area is seen to reduce by 9% as a result of this optimization is also applied to applied to 3-D ICs .

5.CHALLENGES FOR 3-D INTEGRATION

A) THERMAL ISSUES IN 3-D ICs

An extremely important issue in 3-D ICs is heat dissipation. Thermal effect s are already known to significantly impact interconnected /device reliability and performance in high-performance 2-D ICs. The problem is expected to be exacerbated by the reduction in chip size, assuming that same power generated in a 2-D chip will now be generated in a smaller 3-D chip, resulting in a sharp increase in the power and density Analysis of thermal problems in 3-D circuits is therefore necessary to comprehend the limitations of this technology and also to evaluate the thermal robustness of different 3-D technology and design options.

It is well known that most of the heat energy in integrated circuits arises due to transistor switching. This heat energy is typically conducted through the silicon substrate to the  package and then to the ambient by a heat sink .With multi layer device designs,  devices in the upper layer will also generate a significant fraction of the heat .Furthermore, all the active layers will be insulated from each other by layers of dielectrics (LTO, HSQ, polyamide, etc.) which typically have much lower thermal conductivity than Si .Hence ,the heat dissipation issue can become even more acute for 3-D ICs and can cause degradation in device performance ,and reduction in chip reliability due to increased junction leakage, electro migration failures ,and by accelerating other failure mechanisms.

B)   RELIABLITY ISSUES IN 3-D ICs

Three dimensional IC s will possibly introduce some new reliability problems. These reliability issues may arise due to the electro thermal and thermo mechanical effects between various active layers and the interfaces between the active layers, which can also influence existing IC reliability hazards such a electro migration and chip performance. Additionally, heterogeneous integration of technologies using 3-d architecture will increase the need to understand mechanical and thermal behavior of new material of new material interfaces and thin film material thermal and mechanical properties.

6. OVERVIEW OF 3-D IC TECHNOLOGY

1) BEAM   RECRYSTALIZATION

A very popular method of fabricating a second active layer (Si) on top of an existing substrate (oxidized Si wafer )is to deposit polysilicon and fabricate thin film transistors (TFT). To enhance the performance of such transistors ,an intense laser  or electron beam is used to induce recrystalisation  of the polysilicon film to reduce or even eliminate most of the grain boundaries.

Advantage

1.      MOS on transistors fabricated on  polysilicon exhibit very low surface mobility values [of the order of 10 cm/Vs].

2.      MOS transistors  fabricated on polysilicon have high threshold voltages (several  volts) due to the high density of  surface states (several 10 cm ) present at the grain boundaries.

Disadvantage

1.      This technique, however, may not be very practical for 3-D devices because of the high temperature involved during melting of the polysilicon.

2.      Difficulty in controlling the grain size variations.

2) SILICON EPITAXIAL GROWTH

Another technique for forming additional Si layers is  to etch a hole in a passivated wafer and epitaxially  grow a single crystal Si seeded from open window in the ILD.  The Si crystal grows vertically and then laterally to cover the ILD.

Advantage:

1.      The quality of devices fabricated on these  epitaxial layer can be as good as those fabricated underneath on the seed wafer surface, since the grown layer is single crystal with few defects.

Disadvantage

1.      The high temperatures involved in this process cause significant degradation in the quality of devices on lower layers.

2. PROCESSED WAFER BONDING:

An attractive alternative is to bond two fully processed wafers on which devices are fabricated on the surface ,including some interconnects, such that the wafers completely overlap.Interchip  vias are etched to electrically connect both wafers after metallization and prior to the bonding process at 400 degree Celsius. For applications where each chip is required to perform independent processing before communicating with it’s neighbor , this technology can prove attractive .

Advantage

1.      Devices on all active levels have similar electrical properties.

2.      Since all chips can be fabricated separately and later bonded ,there is independence of processing temperature.

Disadvantage

1.      The lack of precision restricts the interchip communication to global metal lines.

3. SOLID PHASE CRYSTALLIZATION (SPC)

In this technique, a layer of amorphous Si is crystallized on top of the lower active layer devices. The amorphous film is randomly crystallized to form a polysilicon film. Device performance can be enhanced by eliminating the grain boundaries in the polysilicon film. For this purpose ,local crystallization can be induced using low temperatures processes (<600C) such as using patterned seeding of germanium . In this method, Ge seeds implanted in narrow patterns made on amorphous Si can be used to included lateral crystallization. This results in the formation of small islands, which are nearly single crystal. CMOS transistors can then be fabricated within these islands to give SOI like performance.

Advantages

1.      This technique offers flexibility of creating multiple active layers

2.      This is a low temperature technique

B) VERTICAL INTERLAYER INTERCONNECT TECHNOLOGY  OPTIONS

There is direct relation between improved chip performance and increased utility of VILICs. It is therefore important to understand how to connect different active layers with a reliable and compatible process. Upper layer processing needs to be compatible with metal lines underneath connecting lower layer devices and metal layers. With Cu technologies, this limits the processing temperatures to <450 c for upper layers. Otherwise , Cu diffusion through barrier layers , and the reliability and thermal stability of material interfaces can degrade significantly. Tungsten is a refractory metal that can be used to withstand higher processing temperatures, but it has higher resistivity. Current via technology can also be employed to achieve VILIC functionality. The underlying assumption here requires that interlayer gates are interconnected using regular horizontal metal wires and vias, while interlayer interconnects can be VILICs connecting the wiring network for each layer.

Recently, interlayer (VLIC)metallization schemes for 3-d ICs have been demonstrated using direct wafer bonding. These techniques are based on the bonding of two wafers with their active layers connected through vias ,which serve as VILICs .

One method is based on the bonding of a thinned top wafer to a bottom wafer with a organic adhesive layer of polyamide in between.

Interchip vias are etched through the ILD(inter level dielectric ),the thinned top silicon wafer and through the cured adhesive layer ,with an approx depth of  20 mm  prior to the bonding process .the interconnect chip via made of chemical wafer depositor (CVD).        Tin liner and CVD-W plug provides a vertical interconnect (VILIC)between the upper most metallization levels of both layers . the bonding between the two wafers is done using a  flip-chip bonder with split beam optics at a temperature of 400 degree Celsius

A second technique realizes on the thermo compression bonding between the metal parts  in each wafer.

In this method, Cu-Ta pads on both wafers save as electrical contacts between the interchips via on the top thinned silicon wafer and  the upper most interconnects on the bottom silicon wafer.  The Cu-Ta pads can also function as small bond pads for wafer bonding. Additionally, dummy metal patterns can be made to increase the surface area for wafer bonding. The Cu-Ta bilayer  pads with a combined thickness of  700 nm are fused together by applying a compressive force at 400 degree Celsius. This technique offers the advantage of a metal –metal interface that will lower the interface thermal resistance between the two wafers (and, hence, provide better conduction) and can be beneficial as a partial ground plane for lowering the electromagnetic effects.

7. PRESENT SCENARIO OF THE 3-D IC INDUSTRY

Many companies are working on the 3-D chips ,including  groups at Massachusetts institute of technology (MIT),international business machines(IBM). Rensselar Polytechnic  and SUNY Albany  are also doing research on techniques for bonding conventional chips together to form multiple layers .whichever approach ultimately wins ,the multilayer chip building technology opens up a whole new world of design .

However ,the Santa Clara, California US  based startup company matrix semiconductor will bring the first multilayer chip to the market ,while matrix’s techniques will not likely result in more computing power ,they will produce cheaper chips for certain applications, like memory used in digital cameras , personal digital assistants ,cellular phones ,hand held gaming devices ,etc .matrix has adapted the technology developed for making flat –panel liquid crystal displays to build chips with multilayer of circuitry.

The company’s first products will be memory chips called 3-Dmemory, for consumer electronics like digital cameras and audio players. current flash memory cards for such devices are rewritable but expensive .however the newly produced chips will cost ten times less, about as much as an audio tape or a roll of film, but will only record information once. The cost is so largely because the stacked chips contain the same amount of circuitry as flash cards but use a much smaller area of the extremely expensive silicon wafers that form the basis for all silicon chips. The chips will also offer a permanent record of the  images and sounds users record. The amount of computing power the company can ultimately build in to its chips could be limited .the company hopes to eventually build chips for cell phones, or low performance micro processors like those found  in appliances; such chips would be about one tenth as expensive as current ones.

The patent technology opens up the ability to build ICs in three dimensions- “up” as well as “out” in the horizontal directions as in the case now with conventional chip designs. The result is a ten fold increase in the potential no of bits on a silicon die, according to the company .moreover,  the 3-D circuits can be produced with todays standard semiconductor materials, fab equipments and processors the 3-D memory will be used in memory devices which will be marketed under well known brand names for portable electronics devices, including digital cameras digital audio players,  games, PDAs and archival digital  storage .the 3-D memory can also be used for pre recorded content such as music, electronics books, digital maps, games, and reference  guides.

8. ADVANTAGES OF 3-D MEMORY

Disks are inexpensive, but they requires drives that are expensive bulky ,fragile and consume a lot of battery power .   Accidentally dropping a  drive or scratching a disk can cause significant  damage and the potential loss of valuable pictures and data.  Flash and other non volatile memories are much more rugged, battery efficient compact and require no bulky drive technologies .  Dropping them is not a problem they are however much more expensive.  Both require the use of a pc.

The ideal solution is  a 3-D  memory that leverages all the benefits of non volatile media, costs as little as a  disk, and is as convenient as 35 mm film and audio tape.

9. APPLICATIONS

Portable electronic digital   cameras, digital audio players, PDAs, smart cellular phones, and handheld gaming devices are among the fastest growing technology market for both business and consumers. To date, one of the largest constraints to growth has been affordable storage, creating the marketing opportunity for ultra low cost internal and external memory. These applications share characters beyond rapid market growth.

Portable devices all require small form factors ,battery efficiency, robustness, and reliability. Both the devices and consumable media are extremely price sensitive with high volumes coming only with the  ability to hit low price points. Device designers often trade application richness to meet tight cost targets. Existing mask ROM and NAND flash non volatile technology force designers and product planners to make the difficult choice between low cost or field programmability and flexibility. Consumers value the convenience and ease of views of readily available low cost storage. The potential to dramatically lower the cost of digital storage weapons many more markets than those listed above. Manufacturers of memory driven devices can now reach price points previously inaccessible and develop richer, easier to use products.

10. FUTURE OF THE 3-D  IC INDUSTRY

Matrix is working with partners including Microsoft Corp, Thomas  Multimedia, Eastman Kodak and Sony Corp. three product categories are planned: bland memory cards: cards  sold  preloaded with content, such as software or music ; and standard memory packages, for using embedded applications such as PDAs and set-top boxes .

Thomson electronics, the European electronic giant, will begin to incorporate 3-D memory chips from matrix semiconductor in portable storage cards, a strong endorsement for the chip start –up.

Thomson multimedia will incorporate the 3-D memory in memory cards  that cane be used to store digital photos or music. Although the cards plug into cameras Thomson is also working on card readers that will allow consumers to view digital photos on a television. The Thomson /matrix cards price makes the difference from completing flash cards from Sony and Toshiba .the 64 MB Thomson card will cost about as much as camera film does today. to further strengthen the relationship with film ,the cards will be sold under the name Technicolor Digital Memory Card.

Similar flash memory cards from other companies cost around Rs.1900 or more-though consumers can erase and rerecord data on them, unlike the matrix cards.            As a result of their price, consumers buy very few of them. Thomson, by contrast , expects to market its write-once cards in retail outlet such as  Wal-Mart.

The first Technicolor cards will offer 64 MB of memory; version with 128 MB and 192 MB will appear later. The first 3-D chips will contain 64 MB. Taiwan Semiconductor Manufacturing Co. is producing the chips on behalf of matrix.

11. CONCLUSION

The 3 D memory will just the first of a new generation of dense, inexpensive chips that promise to make digital recording media both cheap and convenient enough to replace the photographic film and audio tape. We can understand that 3-D ICs  are an attractive chip architecture, that can alleviate the interconnect related problems such as delay and power dissipation and can also facilitate integration of heterogeneous technologies in one chip. The multilayer chip building technology opens up a whole new world of design like a city skyline transformed by skyscrapers, the world of chips may never look at the same again.

12. REFERNCES

1.                  Proceedings of the IEEE, vol 89,no 5,may 2001:

(a)   Jose E Schutt-Aine , sung-Mo Kang,

“Interconnections –addressing the next challenge of IC technology” at     page 583

(b) Robert h Have Mann, James A Hutch by,

“High performance interconnects: an integration overview” at page 586.

(c) Kaustav Banerjee, Shukri J Souri, Pawan Kapur and Krishna C Sara swath           3-D ICs: a novel chip design for improving deep sub micrometer    interconnect performance and  Soc integration at page 602.

2. Electronics today June 2002.

ABSTRACT

The unprecedented   growth of   the computer and the Information technology industry is demanding Very Large Scale Integrated (VLSI) circuits with increasing functionality and performance at minimum cost and power dissipation. VLSI circuits are being aggressively scaled to meet this Demand, which in turn has some serious problems for the semiconductor industry.

Additionally heterogeneous integration of different technologies in one single chip (SoC) is becoming increasingly desirable, for which planar (2-D) ICs may not be suitable.

3-D ICs are an attractive chip architecture that can alleviate the interconnect related problems such as delay and power dissipation and can also facilitate integration of heterogeneous technologies in one chip (SoC).  The multi-layer chip industry opens up a whole new world of design. With the Introduction of 3-D ICs, the world of chips may never look the same again.

CONTENTS

1.  Introduction

2.  Motivation for 3-D ICs

3.  Scope of this study

4.  Area and performance estimation of 3-D ICs

5.  Challenges for 3-D Integration

6.  Overview of 3-D IC technology

7.  Present scenario of the 3-D IC industry

8.  Advantages of 3-d memory

9.  Applications of 3-D ICs

10.  Future of 3-D IC industry

11.  Conclusion

12.  Reference

Pick up any newspaper today and it is a safe bet that you will find an article somewhere relating to mobile communications. If it is not in the technology section it will almost certainly be in the business section and relate to the increasing share prices of operators or equipment manufacturers, or acquisitions and take-overs thereof. Such is the pervasiveness of mobile communications that it is affecting virtually everyone’s life and has become a major political topic and a significant contributor to national gross domestic product (GDP).

The major driver to change in the mobile area in the last ten years has been the massive enabling implications of digital technology, both in digital signal processing and in service provision. The equivalent driver now, and in the next five years, will be the all pervasiveness of software in both networks and terminals. The digital revolution is well underway and we stand at the doorway to the software revolution. Accompanying these changes are societal developments involving the extensions in the use of mobiles. Starting out from speech-dominated services we are now experiencing massive growth in applications involving SMS (Short Message Service) together with the start of Internet applications using WAP (Wireless Application Protocol) and i-mode. The mobile phone has not only followed the watch, the calculator and the organiser as an essential personal accessory but has subsumed all of them. With the new Internet extensions it will also lead to a convergence of the PC, hi-fl and television and provide mobility to facilities previously only available on one network.

The development from first generation analogue systems (1985) to second generation (2G) digital GSM (1992) was the heart of the digital revolution. But much more than this it was a huge success for standardisation emanating from Europe and gradually spreading globally.

However, world-wide roaming still presents some problems with pockets of US standards IS-95 (a code division multiple access [CDMA] rather than a time division multiple access [TDMA] digital system) and IS- 136 (a TDMA variant) still entrenched in some countries. Extensions to GSM (2G) via GPRS (General Packet Radio Service) and EDGE (Enhanced Data rates for GSM Evolution) (E-GPRS) as well as WAP and i-mode (so called 2.5G) will allow the transmission of higher data rates as well as speech prior to the introduction of 3G.

Mobile systems comprise a radio access together with a supporting core network. In GSM the latter is characterised by MAP (Mobile Applications Protocol), which provides the mobility management features of the system.

GSM was designed for digital speech services or for low bit rate data that could fit into a speech channel (e.g. 9.6kbit/s). It is a circuit rather than a packet oriented network and hence is inefficient for data communications. To address the rapid popularity increase of Internet services, GPRS is being added to GSM to allow packet (Internet Protocol [IP]) communications at up to about 100kbit/s.

Third generation (3G) systems were standardised in 1999. These include IMT-2000 (International Mobile Telecommunications 2000), which was standardised within ITU-R and includes the UMTS (Universal Mobile Telecommunications System) European standard from ETSI (European Telecommunications Standards Institute), the US derived CDMA 2000 and the Japanese NTT DoCoMo W-CDMA (Wideband Code Division Multiple Access) system. Such systems extend services to (multirate) high-quality multimedia and to convergent networks of fixed, cellular and satellite components. The radio air interface standards are based upon W-CDMA (UTRA FDD and UTRA TDD in UMTS, multicarrier CDMA 2000 and single carrier UWC-136 on derived US standards). The core network has not been standardised, but a group of three—evolved GSM (MAP), evolved ANSI-41 (from the American National Standards Institute) and IP-based— are all candidates. 3G is also about a diversity of terminal types, including many non-voice terminals, such as those embedded in all sorts of consumer products. Bluetooth (another standard not within the 3G orbit, but likely to be associated with it) is a short-range system that addresses such applications. Thus services from a few bits per second up to 2Mbit/s can be envisioned.

For broadband indoor wireless communications, standards such as HIPERLAN 2 (High Performance Local Area Network—ETSI’s broadband radio access network [BRAN]) and IEEE 802.lla have emerged to support IP based services and provide some QoS (quality of service) support. Such systems are based on orthogonal frequency division multiplexing (OFDM) rather than CDMA and are planned to operate in the 5GHz band.

Whereas 2G operates in 900 and 1800/1900MHz frequency bands, 3G is intended to operate in wider bandwidth allocations at 2GHz. These new frequency bands will provide wider bandwidths for some multimedia services and the first allocations have been made in some countries via spectrum auctions (e.g. in the UK, Holland and Germany) or beauty contests (in France and Italy). The opportunity has also been taken to increase competition by allowing new operators into the bands as well as extending existing operator licences. These new systems will comprise microcells as well as macrocells in order to deliver the higher capacity services efficiently. 3G and 2G will continue to coexist for some time with optimisation of service provision between them. Various modes of delivery will be used to improve coverage in urban, suburban and rural areas, with satellite (and possibly HAPS—high altitude platform stations) playing a role.

The story of the evolution of mobile radio generations is summed up in Fig. 1.

Already, as we move from 2G to 3G the convergence of communications and computing is central to the realisation of the new generation of services and applications. Digital technology enables dynamic adaptation of systems, and intercommunicating software embedded in networks and terminals allows efficient control of the new networks. This is accentuated as we move from 3G to 4G, extending the range and bit rate of services and bringing about the convergence of fixed, mobile and broadcast networks, service provision and terminal types.

This paper introduce the basic ideas and thinking behind the second phase research programme (1999-2003) of the UK’s Virtual Centre of Excellence in Mobile and Personal Communications (Mobile VCE) in the form of ‘visions for 4G’. A Visions Group has been set up to produce and maintain an evolving picture of 4G and to communicate these ideas down to the work areas and researchers. The aim is to provide an umbrella vision to harmonise the research work in the various areas.

The next section explain the limitations of 3G systems and derive the drivers for 4G. The subsequent sections present ‘the 4G vision’ and some of the research challenges that this presents. The approach that is taken here is one of developing a technical vision. However it is based upon likely user scenarios that have been developed within the Mobile VCE

Limitations of 3G and drivers for 4G

From its basic conception to the time of roll-out took around ten years for 2G; a similar period will apply to 3G, which will commence service in 2001/2 and reach full deployment by 2005. Thus by 2010 it will be time to deploy 4G networks and, working backwards with the ten year cycle, it is clear that the year 2000 is appropriate to start with visions for 4G and a research programme aimed at the key issues. The Mobile VCE’s second phase research programme has been constructed to meet this aim.

The starting point was to look at current trends. Here we see a phenomenal growth in mobiles with an estimated global user base that will exceed one billion by 2003. Already mobile communications exceed fixed communications in several countries and it is foreseen that mobile communications will subsume fixed by 2010 (fixed—mobile convergence will be complete). Currently short messaging is booming, especially among the younger generation, with averages of upwards of 100 messages per month dominating monthly bills. Business take-up of SMS via information services is also increasing and providing a start for mobile e-commerce, but this is currently very much limited by the bit rates available. This will be improved with the introduction of GPRS.

In Europe the WAP system (using Wireless Markup Language—WML) has been slow to gain market ground; in contrast, in Japan NTT DoC0oMo’s ‘i-mode’ system had over 10 million subscribers by summer 2000 and is picking up 50000 new customers per day. Customers are already browsing the Internet, exchanging e-mail, conducting banking and stock transactions, making flight reservations and checking news and weather via HTML- based (Hyper Text Markup Language) text information on their phones. Java is expected to be available on i-mode phones soon, allowing the download of agents, games etc. and the introduction of location-based services. In Japan, the number of net phones has now passed the number of wired Internet customers and is setting the trend that others will surely follow when 3G opens up more bandwidth and improved quality.

Thus 3G will provide a significant step in the evolution of mobile personal communications. Mobility appears to be one of the fundamental elements in the evolution of the information society. As service provision based on ‘network centric’ architectures gradually gives way to the ‘edge-centric’ architectures, access is needed from more and more places at all times. But can 3G deliver?

It is true that 3G can support multimedia Internet-type services at improved speeds and quality compared to 2G. The W-CDMA based air-interface has been designed to provide improved high-capacity coverage for medium bit rates (384 kbit/s) and limited coverage at up to 2Mbit/s (in indoor environments). Statistical multiplexing on the air also improves the efficiency of packet mode transmission. However, there are limitations with 3G as follows:

• Extension to higher data rates is difficult with CDMA due to excessive interference between services.
• It is difficult to provide a full range of multirate services, all with different QoS and performance requirements due to the constraints imposed on the core network by the air interface standard. For example, it is not a fully integrated system.

In addition, the bandwidth available in the 2GHz bands allocated for 3G will soon become saturated and there are constraints on the combination of frequency and time division duplex modes imposed by regulators to serve different environments efficiently.

By the year 2010, one of the key enabling technology developments will be embedded radio—the widespread availability and use of the $1 radio chip, which will evolve from short-range wireless developments such as Bluetooth. Embedded radio will eventually become as common as embedded microprocessors are today, with perhaps 50 such devices in the typical home, the user being mostly unaware of their presence. As they interact, in response to the user arriving home for example, they will form a home area network (HAN). Similarly, such devices will be present in large numbers in vehicles (the vehicular area network, or VAN), in personal belongings (the personal area network, or PAN), in the public environment, etc. Such chips will serve as a means of short-range communication between objects and devices, offering capabilities for monitoring and control, in most cases without the knowledge or intervention of the user.

As a person moves between these environments such short-range links will allow their personal profiles and preferences to move with them, with the hotel room automatically configuring itself to their personal preferred temperatures, TV channels/interests, lighting etc. However, the integration of such links with wide-area mobile access will enable far more powerful service concepts, as mobile agents access this pervasive network of sensors and access information on the user’s behalf to perform and even pre-empt their needs and wishes.

In the 1G to 2G transition, as well as a transition from analogue to digital we saw a mono-service to multi-service transition. From 2G to 3G, as well as a mono-media to multimedia transition we are also seeing a transition from person-to-person to person-to-machine interactions, with users accessing video, Internet/intranet and database feeds. The 3G to 4G transition, supported by such technologies, will see a transition towards a pre-dominance of automated and autonomously initiated machine-to-machine interactions.

Such developments will of course be accompanied by ongoing evolution of already anticipated 3G services, such as:

• send/receive e-mail
• Internet browsing (information)
• on-line transactions (e-business)
• location-dependent information
• company database access
• large-file transfer.

These services in themselves represent an increase in requirements for accessing information, for business and commercial transactions, as well as for a raft of new location-dependent information services, all including significantly higher bit-rate requirements. There is a requirement for a mixture of unicast, multicast and broadcast service delivery with dynamic variation between application services both spatially and temporally. Above all, there is a demand for ease of user access and manipulation, with minimal user involvement—complexity hidden from the user—and intelligence to learn and adapt with use.

From the above it will be seen that 4G will need to be highly dynamic in terms of support for:

• the users’ traffic
• air interfaces and terminal types
• radio environments
• quality-of-service types
• mobility patterns.

4G, then, must itself be dynamic and adaptable in all aspects, with built-in intelligence. Thus a ‘software system’ rather than a hard-and-fixed physical system is indicated. Integration, needed to reflect the convergence issues already mentioned, is also a key to 4G, in particular integration of the radio access and the core network elements, which must be designed as a whole rather than segmented as in the past. Key drivers to 4G will be:

• a multitude of diverse devices (distributed, embedded, wearable, pervasive)
• predominance of machine-to-machine communications
• location-dependent and e-business applications
• the extension of IF protocols to mobility and range of QoS
• privacy and security
• dynamic networking and air-interfaces
• improved coverage mechanisms
• improved and dynamic spectrum usage.

4G visions mapping to research topics

The Mobile VCE vision for 2010 is embodied in the five key elements shown in Fig. 2 and detailed as follows:

• Fully converged services: Personal communications, information systems, broadcast and entertainment will have merged into a seamless pool of content available according to the user’s requirement. The user will have access to a wider range of services and applications, available conveniently, securely and in a manner reflecting the user’s personal preferences.
• Ubiquitous mobile access: The dominant mode of access to this pool of content will be mobile, accounting for all voice communications, the majority of high-speed information services, and a significant proportion of broadcast and entertainment services. Mobile access to commercial and retail services will be the norm, replacing current practices in most cases.
• Diverse user devices: The user will be served by a wide variety of low-cost mobile devices to access content conveniently and seamlessly. These devices will commonly be wearable—in some cases disposable— and will normally be powered independently of the mains. Devices will interact with users in a multi sensory manner, encompassing not only speech, hearing and sight but also the other human senses, and biological and environmental data pertinent to the application. Special devices tailored for people with disabilities will be common place
• Autonomous networks: Underlying these systems will be highly autonomous adaptive networks capable of self-management of their structure to meet the changing and evolving demands of users for both services and capacity. Efficient and cost-effective use of the radio spectrum will be an essential element of their operation, and here, too, autonomy and self- management will be the norm.
• Software dependency: Intelligent mobile agents will exist throughout the networks and in user devices, and will act continually to simplify tasks and ensure transparency to the user. These mobile agents will act at all levels, from managing an individual user’s content preferences to organising and reconfiguring major elements of networks.

Research challenges

Analysis of the underlying technical challenges raised by the above vision and its five elements has produced three research areas: Networks and services, Software based systems, Wireless access. These form the basis of the Mobile VCE Phase 2 research programme.

Networks and services

The aim of 3G is ‘to provide multimedia multirate mobile communications anytime and anywhere’, though this aim can only be partially met. It will be uneconomic to meet this requirement with cellular mobile radio only. 4G will extend the scenario to an all-IP network (access + core) that integrates broadcast, cellular, cordless, WLAN (wireless local area network), short-range systems and fixed wire. The vision is of integration across these network—air interfaces and of a variety of radio environments on a common, flexible and expandable platform — a ‘network of networks’ with distinctive radio access connected to a seamless IP-based core network a (Fig. 3).

The functions contained in this vision will be:

• a connection layer between the radio access and the IP core including mobility management
• internetworking between access schemes — inter and intra system, handover, QoS negotiations, security and mobility
• ability to interface with a range of new and existing radio interfaces

A vertical view of this 4G vision (Fig. 4) shows the layered structure of hierarchical cells that facilitates optimisation for different applications and in different radio environments. In this depiction we need to provide global roaming across all layers.

Both vertical and horizontal handover between different access schemes will be available to provide seamless service and quality of service.

Network reconfigurability is a means of achieving the above scenario. This encompasses terminal reconfigurability, which enables the terminal to roam across the different air interfaces by exchanging configuration software (derived from the software radio concept). It also provides dynamic service flexibility and trading of access across the different networks by dynamically optimising the network nodes in the end-to- end connection. This involves reconfiguration of protocol stacks, programmability of network nodes and reconfigurability of base stations and terminals.

The requirement is for a distributed reconfiguration control. Fig. 5 demonstrates both internal node and external network reconfigurability.

For internal reconfiguration the functionality of the network nodes must be controlled before, during and after reconfiguration and compliance to transmission standards and regulations must be facilitated.

External reconfiguration management is required to monitor traffic, to ensure that the means for transport between terminals and network gateways (or other end points) are synchronised (e.g. by conforming to standards) and to ensure that the databases/content servers needed for downloadable reconfiguration software are provided.

The research challenges are to provide mechanisms to implement internal and external configuration, to define and identify application programming interfaces (APIs) and to design mechanisms to ensure that reconfigured network nodes comply with regulatory standards.

An example of evolved system architectures is a combination of ad hoc and cellular topologies. A ‘mobile ad hoc network’ (MANET) is an autonomous system of mobile routers (and connected hosts) connected by wireless links. The routing and hosts are free to move randomly and organise themselves arbitrarily; thus the network wireless topology can change rapidly. Such a network can exist in a stand-alone form or be connected to a larger internet (as shown in Fig. 6).

In the current cellular systems, which are based on a star-topology, if the base stations are also considered to be mobile nodes the result becomes a ‘network of mobile nodes’ in which a base station acts as a gateway providing a bridge between two remote ad hoc networks or as a gateway to the fixed network. This architecture of hybrid star and ad hoc networks has many benefits; for example it allows self-reconfiguration and adaptability to highly variable mobile characteristics (e.g. channel conditions, traffic distribution variations, load-balancing) and it helps to minimise inaccuracies in estimating the location of mobiles.

Together with the benefits there are also some new challenges, which mainly reside in the unpredictability of the network topology due to mobility of the nodes; this unpredictability, coupled with the local-broadcast capability, provides new challenges in designing a communication system on top of an ad hoc wireless network. The following will be required:

• distributed MAC (medium access control) and dynamic routing support
• wireless service location protocols
• wireless dynamic host configuration protocols
• distributed LAC and QoS-based routing schemes.

In mobile IP networks we cannot provide absolute quality-of-service guarantees, but various levels of quality can be ‘guaranteed’ at a cost to other resources. As the complexity of the networks and the range of the services increase there is a trade-off between resource management costs and quality of service that needs to be optimised. The whole issue of resource management in a mobile IP network is a complex trade-off of signaling, scalability, delay and offered QoS.

As already mentioned, in 4G we will encounter a whole range of new multirate services, whose traffic models in isolation and in mixed mode need to be further examined. It is likely that aggregate models will not be sufficient for the design and dynamic control of such networks. The effects of traffic scheduling, MAC and CAC (connection admission control) and mobility will be required to devise the dimensioning tools needed to design 4G networks.

Software systems

We have already seen in the previous subsection that to effect terminal and network node reconfigurability we need a middleware layer. This consists of network intelligence in the form of object-oriented distributed processing and supporting environments that offer the openness necessary to break down traditional boundaries to interoperability and uniform service provision. The mobile software agent approach is an especially important building block as it offers the ability to cope with the complexities of distributed systems. Such building blocks may reside at one time in the terminal and then in the network; or they may be composed of other objects that themselves are mobile. Within the mobile system there exists a range of objects whose naming, addressing and location are key new issues. A further step in this development is the application of the Web-service-model rather than the client/server principle; recent industry tendencies show a shift towards this paradigm and XML (extensible Markup Language) is seen as the technology of the future for Web-based distributed services. However this technology has yet to prove its scalability and suitability for future application in mobile networks.

In addition to the network utilities there will be a range of applications and services within 4G that also have associated with them objects, interfaces (APIs) and protocols. It is the entirety of different technologies that underlies the middleware for the new 4G software system.

The ‘killer application’ for 4G is likely to be the personal mobile assistant (PMA)—in effect the software complement to the personal area network—that will organise, share and enhance all of our daily routines and life situations. It will provide a range of functions including:

• Ability to learn from experiences and to build on personal experiences, i.e. to have intelligence
• Decision capability to organise routine functions with other PMAs and network data bases, e.g. diary, travel arrangements, holidays, prompts (shopping, haircut, theatre, birthdays, etc.)
• A range of communication modes: voice, image (with image superimposition via head-up displays such as glasses or retinal overlays), multiparty meetings (including live action video of us and our current environment), etc.
• Provision of navigation and positioning information and thus of location-dependent services:
• Detecting and reporting the location of children, pets and objects of any sort
• Vehicle positioning and route planning, auto pilot and pedestrian warnings
• Automatic reporting of accidents (to insurance companies, rescue services and car dealers)
• Knowledge provision via intelligent browsing of the Internet
• E-business facilities for purchasing and payment
• Health monitoring and provision of warnings
• Infotainment: music, video and, maybe, virtual reality

Of course the key to all this is ‘mobility’—we need to have the ‘PMA’ whenever and wherever we are, and this places additional complexity on network and service objects and the agents that process them.

Specifically we need to consider what the metrics are that determine which objects follow the user. Some objects can move anywhere; others can move in some directions or within a constrained area. If they can move, how will the existing service determine if resources are available to support them in their new (temporary) home? Will they still be able to function? What kind of computing architecture and middleware platforms will be capable of supporting thousands, perhaps millions, of such objects?

Aspects of security pervade the whole of this area. Rules of authentication, confidentiality, scalability and availability must now be applied to objects that are continuously mobile. A whole set of conditions that are valid at one time and place maybe invalid if transferred to another. Integrity and correctness issues must be considered when mechanisms that support applications are used in practice in the presence of other; distributed algorithms. For issues such as liveness, safety and boundedness—consistency, isolation and durability— execution semantics need to be evidenced for extension to the mobile environment.

Distributed management tools, in a complementary way, will allow a certain level of monitoring (including collection of data for analysis), control and troubleshooting. The management tools currently available do not encompass mobility efficiently and hence this is another important area of research.

The aim of the research in this area is to develop tools that can be used in 4G software systems. The following specific scenarios are being addressed in order to focus the issues:

• E-commerce, including microtransactions, share trading and internal business transactions
• Home services, ranging from terminal enhancements (e.g. enhancing the display capabilities by using the TV screen as a display unit for the terminal) to security systems and housekeeping tasks
• Transportation systems: Itinerary support, ticketing and location services are to be targeted in this area.
• Infotainment on the move: This will demonstrate the need for software and terminal reconfiguration and media-adaptation.
• Telemedicine and assistance services: Emergency team support, remote/virtual operations and surveillance of heart patients are possible stages for this scenario.

This list of scenarios can be expanded arbitrarily and also into non-consumer areas (i.e. military and emergency services), however the preconditions for service delivery and demands on the network infrastructure remain the same: they will have to be adaptable to meet the user- requirements current in 2010. Support for these scenarios may be given by intelligent agents, which may represent the terminal within the network to manage the adaptations or customisations of the communication path. On an application or service layer they may additionally be used to complete business transactions for the user (e.g. booking a theatre ticket or a flight) or to support other services. Furthermore, distributed software entities (including the variety of models from objects, via agents, to the Web-service model) will encompass management and support for applications and services as well as for user and terminal mobility.

Wireless access

In the previous two sections we have looked at the type of network and the software platforms needed to reconfigure, adapt, manage and control a diversity of multimedia, multirate services and network connections. We have seen that there will be a range of radio access air interfaces optimised to the environments and the service sets that they support. The reconfigurability and the middleware flow through to the wireless access network. The radio part of the 4G system will be driven by the different radio environments, the spectrum constraints and the requirement to operate at varying and much higher bit rates and in a packet mode. Thus the drivers are:

• Adaptive reconfigurability—algorithms
• Spectral efficiency—air interface design and allocation of bandwidth
• Environment coverage—all pervasive
• Software—for the radio and the network access
• Technology—embedded/wearable/low-power/high communication time/displays.

It has been decided within Mobile VCE not to become involved in technology issues or in the design of terminals. This is a large area, which is much closer to products and better suited to industry. The remaining drivers are all considered within the research programme.

It is possible, in principle, to increase significantly the effective bit rate capacity of a given bandwidth by using adaptive signal processing at both the base station and the mobile. In 3G systems adaptive signal processing has been restricted to the base station and so the challenge is to migrate this to the terminal and, most importantly, to make the two ends co-operative. Such techniques require close co-operation between the base and mobile stations in signalling information on channel quality, whilst making decisions and allocating resources dynamically. In addition, the capabilities of both ends of the link must be known reciprocally as the channel varies in both time and space. In order to optimise a link continuously, the wireless network must acquire and process accurate knowledge of metrics that indicate the current system performance, e.g. noise, inter- and intra-system interference, location, movement variations, and channel quality prediction. Such information and its accuracy must be passed to the higher layers of the system protocol that make decisions and effect resource allocation. The emphasis on the base station in 3G systems is obvious as this has the resources, real estate and capacity to implement the spatial—temporal digital signal processing needed for antenna arrays together with advanced receiver architectures. The challenge will be to migrate this to the much smaller terminal via efficient electronics and algorithms that will still allow a range of services and good call time. The availability of individual link metrics can also be used at a network level to optimise dynamically the network radio resources and to produce a self-planning network.

Arguably the most significant driver in the wireless access is the bandwidth availability and usage and whereabouts in the spectrum it will fall. Currently 3G technology is based around bands at 2GHz, but limited spectrum is available, even with the addition of the expansion bands. The higher bit rates envisaged for 4G networks will require more bandwidth. Where is this to be found? The scope for a world-wide bandwidth allocation is severely constrained and, even if this were feasible, the bandwidth would be very limited. The requirements are thus for much more efficient utilisation of the spectrum and, perhaps, new ideas for system co-existence. If the bandwidth is fixed we need to seek a spectrally more efficient air interface and this involves a consideration of various multiple access, modulation, coding, equalisation/interference cancellation, power control, etc. schemes. In view of our previous comments it is clear that all components of this air interface must be dynamically adaptive. As the whole network is to be IP based this will mean extremely rapid adaptation on a burst basis. In 4G systems we need to accomplish this at much higher and variable bit rates as well as in different environments (indoor, outdoor, broadcast, etc.) and in the presence of other adaptive parameters in the air interface. In time-domain systems equalisers would need to be adaptive and this raises questions of complexity. For CDMA, systems could use multicodes and adaptive interference cancellation, which again raise complexity issues. Alternatively one could move to OFDM-like systems (as in WLANs), which offer some reduction in complexity by operating in the frequency domain but raise other issues, such as synchronisation. The choice of the air interface’s multiple access scheme and adaptive components will need to be based upon the ease of adaptation and reconfigurability and on the complexity. There are also significant research challenges in this area of flexible advanced terminal architectures that are not rooted solely in physical layer problems.

A further aspect of spectrum efficiency relates to the way in which regulators allocate bandwidth. The current practice of exclusive licensing of a block of spectrum is arguably not the most efficient. It would be much more efficient to allow different operators and radio standards to co-exist in the same spectrum by dynamically allocating spectrum as loading demands. Indeed, the higher bit-rate services may need to spread their requirements across several segments of spectrum. There would then be a need for a set of rules to govern the dynamic allocation of the spectrum—a self organising set of systems to maximise the use of spectrum and balance the load. Given the degree of co-operation and the processing already envisioned this should be a realistic aim.

A great deal of work on the characterisation of radio environments has already been performed in the 2GHz and 5GHz bands within the first phase of Mobile VCE’s research, and spatial—temporal channel models have been produced. However, 4G systems will incorporate smart antennas at both ends of the radio link with the aim of using antenna diversity in the tasks of canceling out interference and assisting in signal extraction. This implies that direction-of-arrival information, including all multipath components, will be an important parameter in determining the performance of array processing techniques. There is a need to augment models with such data for both the base station and the terminal station. A more open question is where to position the next frequency bands for mobile communications. An early study is needed here in advance of more detailed radio environment characterisations.

Coverage is likely to remain a problem throughout the lifetime of 3G systems. The network-of-networks structure of 4G systems, together with the addition of multimedia, multirate services, mean that coverage will continue to present challenges. We have already seen that the likely structure will be based upon a hierarchical arrangement of macro-, micro- and picocells. Superimposed on this will be the megacell, which will provide the integration of broadcast services in a wider sense. Until now, it has been assumed that satellites would provide such an overlay, and indeed they will in some areas of the world. However, another attractive alternative could be high-altitude platform stations (HAPS), which have many benefits, particularly in aiding integration.

HAPS are not an alternative to satellite communications, rather they are a complementary element to terrestrial network architectures, mainly providing overlaid macro-/microcells for underlaid picocells supported through ground-based terrestrial mobile systems. These platforms can be made quasi- stationary at an altitude around 21—25 km in the stratospheric layer and project hundreds of cells over metropolitan areas (Fig. 7).

Due to the large coverage provided by each platform, they are highly suitable for providing local broadcasting services. A communication payload supporting 3G/4G and terrestrial DAB/DVD air interfaces and spectrum could also support broadband and very asymmetric services more efficiently than 3G/4G or DAB/DVD air- interfaces could individually. ITU-R has already recognised the use of HAPS as high base stations as an option for part of the terrestrial delivery of IMT-2000 in the bands 1885—1980 MHz, 2010—2025 MHz and 2110—2170 MHz in Regions 1 and 3, and 1885—1980 MHz and 2110—2160 MHz in Region 2 (Recommendation ITU-R M (IMT-HAPS)).

HAPS have many other advantages in reducing terrestrial real-estate problems, achieving rapid roll-out, providing improved interface management to hundreds of cells, spectrally efficient delivery of multicast/broadcast, provision of location-based services and, of course, integration. The research challenge is to integrate terrestrial and HAPS radio access so as to enhance spectral efficiency and preserve QoS for the range of services offered.

Software, algorithms and technology are the keys to the wireless access sector. Interplay between them will be the key to the eventual system selection, but the Mobile VCE’s research programme will not be constrained in this way. The aim is to research new techniques which themselves will form the building blocks of 4G.

Conclusion

It is always dangerous to predict too far ahead in a fast- moving field such as mobile communications. Almost by definition the eventual 2010 scene will not match exactly that depicted in the 4G vision described herein. However,  the key elements—fully converged services, ubiquitous mobile access, diverse user devices, autonomous networks and software dependency—will persist. The 4G Vision is a living document which intends to update and amend as time and knowledge progress. It will act as the umbrella vision to a large research programme and place in context the detailed research work that will take place in the various areas. In this respect it will help to continuously steer the research as it progresses and, therefore, to make it more relevant and beneficial.